Technical Debt pattern: Custom off-the-shelf

It was only about 30 minutes into the meeting when the senior developer uttered the dreaded words: “Rewrite”. That was the point where what should have been a simple 6-step upgrade turned into a 9 month nightmare upgrade/rewrite costing us millions with nothing new to show for it and left us with a broken team and a disillusioned business.

A few years earlier the company (a bank) had decided to buy a Business Intelligence (BI) product from Oracle to generate insights into badly performing commercial loans. By now not just a vital part in saving tens of companies a year from bankruptcy, but also played a big part in our regulatory obligations.

Buying the Oracle product had allowed the bank to quickly get a working version up and running and as a result feature requests quickly started pouring in and a team was formed to keep adding features to the BI solution.

A few years later and a month or two before I joined the team Oracle dropped extended support for the version of Oracle BI and after trying to upgrade for a few months the decision was made to cut our losses and rewrite it from scratch.

How did we get into this mess?

Off-the-shelf turned into custom made The decision to buy COTS (Commercial Off-The-Shelf) was made so we could get started quickly, which was (quite literally) invaluable, because not just the amount of money involved, but also to keep regulators happy. But at no point was that decision revisited or underlying assumptions reconsidered. More and more complex features were added until it was no an off-the-shelf product anymore. I have seen this so many times I call it the “Custom Off-The-Shelf” pattern.

More features over keeping up with upgrades

The business and technology teams were not understanding each other. The business cared only about getting the most bang for their buck and when features would be delivered. The technology team in turn was not transparent what the consequences were of those decisions. One major example of that, was that features were consistently prioritized over things like upgrades. This building up of technical debt dramatically decreased the maintainability and the transparency of progress. By the time we were forced to rewrite the application, it was years out of date and Oracle had dropped even extended support.

Unclear coding standards & design guidelines

Most of the features were custom built into the product, and it was not transparent how these features were exactly built into the product. In the bank, the product was seen as off-the-shelf and very specific to this business area, so setting up coding standards was deemed unnecessary. The team was only one of the two teams in the bank that were using this product. And because of the relative low amount of development on this product, the team had never spent much time on getting clear coding standards and design guidelines. Over the years, team members had come and gone, and design decisions had gone with them. Back tracing specific design decisions for the custom built features were very hard..

What should we have done differently?

• Say ‘No’ and be transparent: The Development Team felt pressured to always say ‘Yes’ to all requests of the business, and failed to articulate the impact and consequences of those decisions: increasingly building up technical debt over the years to a point that it was no longer maintainable. Learn to explain the impact of technical debt, and learn to say ‘No’.

• Reassess the buy the decision versus custom built: The reality was that this was not an off-the-shelf product anymore. The custom built features were the majority of the application. We should have reassessed the buy versus custom built decisions regularly.

• Never ask permission to do your job properly: The Development Team is responsible for delivering high quality Increments. As a Development Team Member you should never ask permission to do your job properly. Upgrades, design & coding standards, tests and anything else you need to delivery quality are non-negotiable. Leave technical decisions to the people accountable for technical quality. Do not give that choice away.

To summarize

Buying an off-the-shelf product can be a great solution for your business problem. But as soon as you start customizing, you need to regularly revisit that decision. As in all software development it is vital to be transparent about impact of any decision Product Owners & stakeholders need to make. And most importantly, never ask permission to do your job properly and address that elephant in the room.