Technical Debt: The Silent Killer
October 27, 2021 Alex Woodie
There’s a dangerous malady afflicting companies that rely on the IBM i server to run their business applications. Left undiagnosed and untreated, it can debilitate an IT shop, rendering it unable to take advantage of new opportunities and respond to challenges. It’s called technical debt, and you or an organization you know may be suffering from it as we speak.
Technical debt is a concept that is credited to Ward Cunningham, the American programmer and the co-author of Manifesto for Agile Software Development. “If you develop a program for a long period of time by only adding features but never reorganizing it to reflect your understanding of those features, then eventually that program simply does not contain any understanding and all efforts to work on it take longer and longer,” Cunningham said in a 2009 video on the debt metaphor.
One of the IBM i professionals who has given the concept of technical debt some thought is Chris Burns. The longtime consultant for IBM i-based insurance application vendor GEMKO Information Group, which was acquired by Tri-Delta Resources in April, presented an eye-opening session on the ravages of technical debt during the recent POWERUp 2021 conference that was sponsored by COMMON.
Technical debt, at a high level, is fairly easy to define. “It boils down to . . . accrued liabilities and costs that you weren’t anticipating,” Burns says. “And they accrued because of taking shortcuts or not following best practices.” Avoiding it is something else altogether.
Technical debt can damage your business in several ways: a loss of productivity, punitive vendor fees, unplanned downtime, a need for additional outside services, and cost overruns. It can blow holes in budgets, cause you to miss opportunities, and lead you to fail to take advantage of assets that you own, Burns said.
While technical debt has various ramifications, the causes invariably come down to people not making good decisions, “typically from the illusion that you don’t have to do things the right way, and that there won’t be any consequences if you don’t do things the right way,” Burns said during his POWERUp presentation.
“Sometimes you have information policies that have double standards. ‘Rules for thee but not for me,'” he says. “The boss can skirt around the rules, but other people can’t. That’s a source of technical debt.”
Image source: Chris Burns
Technical debt bares many resemblances to the National Debt, which currently stands at $28.9 trillion, according to the US Debt Clock. Both stem from people making poor decisions, usually to curry favor with a certain group. The debt is passed from one generation to the next, as paying the interest is cheaper than paying off the full debt. But eventually, the full debt will come due. The only question is when.
If you want to avoid getting into debt, then Burns encourages you to avoid certain bad practices, which Burns spelled out in detail.
Lapsed Maintenance
For starters, letting your maintenance coverage lapse is a good way to start building technical debt. “Sometimes upper-level managers don’t like the idea of writing a big check to a software company for your annual license fee or for your support,” he said. “[They think] ‘I’ll just let that lapse and then we’ll save $20,000 or $30,000 a year and I look like a hero. Maybe I’ll get a bonus.’”
Going off support essentially locks you into a certain ERP release, which often limits your ability to upgrade the operating system and the underlying hardware. When you finally realize the error of your ways, the ERP vendor will often require you to pay for all the years you missed, plus a penalty, and then pre-pay for a period of three to five years, Burns said.
“So that’s saving $20,000 year for 10 years might end up cost you half a million or a million dollars,” he said. “And that might end up costing the head of the person who made that decision. And the only way out of that is your checkbook, write a big check with lots of zeros. That’s the only way you’re going to get out of Lapser-ville.”
Another fast track to technical debt involves cloning code. It’s been said that programmers often write only one program in their lives, then clone it 1,000 times. Those are the bad programmers who accumulate technical debt on behalf of their employers.
“The world is filled with one-time programs that end up in the night job in 60 days and suddenly it’s in your daily run,” Burns said. “But even if you only use it once, somehow it hangs around forever in the code library, because nobody wants to get rid of it.”
Cloned Programs
Cloned programs are due to the “Burger King” culture that’s prevalent in IT, where everybody gets to have it their way, Burns said. When the boss asks for special treatment, it’s hard to say no. Instead, you ask if he wants pickles with his cloned bit of code. “People want their own version, grumpy people who always get their way,” he said.
Flame-broiled burgers are not like ERP systems: You cannot always have it your way.
When you come across a bit of cloned code, even if you’re fairly certain it can be removed, you’ll leave it out of fear that it may come to life at some point, or break something downstream, Burns said. The cloned code will proliferate, requiring you to hunt down each and every version of the program when it comes time to update or modernize the program.
The solution to cloned code is a little old-fashioned medicine. Instead of accommodating the Burger King culture, you must learn to say no. Instead of customizing each and every program, you must train your users to utilize the functionality that ships with the system.
“In the most of the sessions here at POWERUp, HA means high availability,” Burns said. “But when you’re talking about technical debt, HA stands for ‘hard ass.’ Somebody has to be the hard ass, the S.O.B. who is going to combat that culture. And it needs to be somebody who wears some stripes, who can enforce some discipline, including discipline on IT to not cave into user whims.”
Hard Coding
Another mark of the indebted programmer is the use of hard coding. This bad habit is more common in shops that write their own software, and it usually has to do with things like customer numbers, company numbers, vendor numbers, user profiles, business unit IDs, and warehouse numbers, Burns said. It’s also more common in IBM i shops with S/36 compatible code, he said.
What worked for the S/36-era Sumerians doesn’t necessarily work for modern RPG.
“Hard coding tends to be done for quick fixes to get around the rules,” he said. “You need to make a tweak or a new trading partner that you brought on, or there’s been kind of a knee jerk policy change in the organization and you’ve got to in and fix the code up a little bit or anywhere where you trying to fit a square peg in a round hole.”
Sometimes hard coding something is unavoidable under tight deadlines. But as soon as there is time, the work should be made right. Technical debt begins to build when the hard coding is allowed to stay. When that bit of code is then cloned and the comments or documentation is lost, then the debt begins to build even faster.
“If you want to reduce and eliminate debt, we really want to try to take conditional statements and instead of make them based hard-coded identifiers, some kind of reusable Boolean expressions that are based on data from the database,” Burns said. “If we can do that, then we can shift the responsibility over to the user community to maintain that.”
Undocumented Code
Not documenting the code is a classic route to incurring technical debt. Back in the old days, technical writers were called upon to document exactly what the code was doing, which made life easier for folks who would work on the application in the future. But like bell-bottom pants and fanny packs, technical writers have gone out of style.
Instead of writing stuff down, the documentation “sits in one person’s head forever and ever and ever, and you hope that person lives a nice long healthy life and doesn’t catch COVID,” Burns said.
The result is that your company literally doesn’t know how its applications work. Your homegrown code may be highly customized and tuned to your specific business model, providing a compelling business advantage, but how it actually works is anybody’s guess.
“So that makes modifications and troubleshooting a lot more difficult if we don’t know how it’s supposed to work,” Burns said. “With modernization, if we don’t know how it’s supposed to work, how are we supposed to modernize it?”
The solution, once again, is a hefty dose of HA. The HA must step up and compel documentation to occur, burns said.
“There’s got to be accountability,” he said. “Too many requests are over the phone or an email. They don’t find their way into a database somewhere or into an official document. . . . This stuff really needs to get online where it’s visible by all.”
Over-customized your code? You get NOTHING!
Over-Customization of Packaged Software
Customizing packaged software is a cardinal sin that can lead to technical debt years down the line. While having a tailored application can provide a competitive advantage on the one hand, it can also work to hold you back.
“The instant you get in there, you’ve broken the warranty seal on the code and so you’ll no longer get support on it because you modified it,” Burns said. “And then of course there’s unintended consequences because the vendor who wrote the software spent years testing it. You went in and you hacked it, and you tested it for maybe 10 minutes.”
Untangling the original source code from the modifications and the comments (if they’ve bothered to document them at all) can be a quite difficult, especially when modifications have been made to previous modifications. The icing on this technical debt cake, though, is when the customer is no longer able to apply new releases of their software package because the software has been overly customized.
“Even if we’re paying this SWMA, even if we’re paying the annual license fee, you know what we get out of all that? If you remember Hogan’s Heroes, it means we get nothing! You pay all that money, and you get nothing! Because you can’t leverage the assets you’re entitled to, which are the upgrades because you’ve over customized.”
Getting Out of Debt
Burns shared some great tips for how to avoid technical debt in the first place: Don’t hard-code or clone your code, be sure you document your changes, and don’t over-customize your code. You’ll also want to use change management software, test your software, and look into service-enablement
But what happens if you find yourself in technical debt already? With some discipline and a lot of hard work, you can rescue yourself. Burns encourages users to start small, stay determined, and keep moving forward. Technical debt tends to build up with older, monolithic programs, so breaking the monolith up into smaller pieces of bite-sized service programs that are well-documented and that fit together in a microservices-based manner is a good way to avoid technical debt from accumulating.
As you work through and modernize the code, a great thing begins to happen.
“A lot of nonproductive code snippets can go away when you use SQL or you use functions to get your data as opposed to some of the older RPG constructs,” Burns said. “As you refit those programs, they start losing weight. That non-productive code starts going away and the programs begin to thin out a little bit to the point where [the] strategic and business reason for the program becomes a lot more obvious.”
RELATED STORIES
The Mod Squad Comes Together to Modernize Old RPG
Planning A Modernization Project? Read This First
Excellent – pretty much nailed it.
One of the better articles I have read in the last couple of years. Most of my Programmer searches in the last 5 years are because some Sr Programmer died or retired. Poor documentation and over customization with no documentation or using the same type of code for every user request seems to be the fault of Programmers not wanting to learn/improve, managers not willing to sell improved systems to management and shops that are understaffed because managers are afraid to ask for more budget. CEO’s, CFO’s need to be aware of technical debt at their companies. Why are some companies still using RPG II programs? Why are companies not keeping up with software releases, not testing their disaster plans? I suggest this article be circulated to your C-level people followed up with an IT Steering Committee meeting. Oh you probably don’t have an IT steering committee!
Often, it is not the programmers, but the management. Management often puts restrictions into place on the modern developers. We have done this way for 40-years, and it still works! Restrictions are put in place from the top. Not from the Developers writing the code.