When Southwest Airlines had to cancel more than 16,700 flights following a winter storm, old crew-scheduling software became the focus of attention.

“This problem -- relying on older or deficient software that needs updating -- is known as incurring technical debt, meaning there is a gap between what the software needs to be and what it is,” wrote Columbia University professor Zeynep Tufekci in a Dec. 31 New York Times opinion piece.

Technical debt stems from aging code as well as new software written in a fast and careless way. “As you might expect, quicker is cheaper,” Tufekci wrote.

Instead of reporting their absence through software, employees were left on hold calling for hours, according to Tufekci.

“When you develop an application, you take certain shortcuts for which you're going to have to pay the price back later on,” explains Olivier Gaudin, cofounder and CEO of SonarSource, which develops open-source software for continuous code quality and security. “You accept that your code is not perfect. You know that it will have a certain cost when you come back to it later. It will be a bit more difficult to read, to maintain or to change.”

Although the crew scheduling software didn’t fail altogether, it couldn’t handle the number of crew and schedule changes, Andrew Watterson, chief operating officer for Southwest Airlines, told the US Senate Committee on Commerce, Science and Transportation on Feb. 9.

“Without updated crew schedules, the crew decision-support software could not reassign crews to solve for flights with crew coverage issues,” Watterson says. “Of the three moving parts in our point-to-point network -- the flight network, the aircraft network, and the crew network -- and as far as our technology is concerned, the disruption primarily revealed a need to add functionality to our crew-scheduling software to solve for a large backlog of broken crew pairings.”

Debrup Ghosh, senior product manager in the Software Integrity Group at Synopsys, notes that scheduling software is a core piece that must stay in tune with other components to keep the company operating properly.

“Scheduling should be a core operational piece for a sophisticated operation like an airline, where all the pieces -- the aircraft, pilots, cabin crew, passengers and baggage -- must come together like a well-tuned orchestra,” Ghosh says.

How Deep Is Technical Debt?

Poor software quality results in steep costs for US organizations of at least $2.41 trillion, up from $1.31 trillion two years ago, according to the report “The Cost of Poor Software Quality in the US: A 2022 Report,” by the Consortium for Information & Software Quality (CISQ) and cosponsored by Synopsys. In addition, the CISQ report found technical debt to be the biggest obstacle to updating existing codebases, costing companies $1.52 trillion.

“Based on my experience, the average tech debt is often 20-25% and, when compounding this across industries and companies, the $1.52 trillion figure doesn’t surprise me,” Ghosh says.

In the case of Southwest Airlines, technical debt cost the company $725 million to $825 million in losses, according to a Jan. 6 filing with the Securities and Exchange Commission.

Technical Debt Hinders Digital Transformation

In a recent survey called “The Path to Digital Transformation: Where IT Leaders Stand in 2023” by IDG company Foundry commissioned by Insight Enterprises, an IT services and solutions provider, technical debt was the No. 3 pain point for large organizations as they aim to carry out digital transformation.

“It ranks high because many systems saddled with this debt are also still systems of high importance to the organization,” says Juan Orlandini, chief architect and distinguished engineer at Insight Enterprises. “You have to keep updating them, keep them running, and provide value. The more debt you incur in a solution, the harder it is to keep it running and/or add capabilities. The insidious thing is that unless you tackle it, the debt continues to accumulate and becomes a bigger challenge over time.”

In addition, close to 40% of respondents named technical debt as one of the top IT infrastructure and operational barriers to digital transformation. The two areas that were constraining digital transformation more were technical skills and knowledge at 45% and budget constraints at 42%.

Meanwhile, 86% of the IT and business decision-makers said technical debt impacted their organization, according to the survey, for which Foundry interviewed 400 senior IT decision-makers at companies with at least 1,500 employees.

“Because such a high percentage of organizations are dealing with technical debt, it follows that technical debt would rank as a top three challenge to digital transformation, coming after skills gaps and budget constraints,” Orlandini says. “Both of the latter are constant challenges. Organizations rarely, if ever, have infinite budget for all of their initiatives -- and technology is changing at such a high pace that we will face skills shortages for the foreseeable future.”

Foundry’s survey found that 43% of the IT decision-makers cited the ability to innovate as a hindrance to digital transformation, 41% mentioned meeting service level agreements (SLAs) and 37% cited avoiding downtime.

“Put simply, technical debt is the enemy of efficiency,” Orlandini says. “It takes the focus away from strategic initiatives. You are spending your resources, time, and energy in maintaining legacy systems rather than in innovation. These legacy systems also become harder and harder to maintain over time -- negatively affecting your ability to meet SLAs.”

What Causes Technical Debt?

For companies accumulating technical debt, they focus on simple IT solutions rather than products for the long term, according to Orlandini. Companies fail to anticipate the scale, use cases, and environmental changes that will occur.

“What was the right answer for the world 10 years ago might not be the right answer for the world today,” Orlandini says.

Ghosh says technical debt stems from poor strategy, a lack of standardization, and failure to allocate resources properly. He explains that companies continue to use old software that require coordination across multiple development teams to introduce new products and features.

“Hence, as companies try to innovate and stay ahead of the competition, they encounter the classic innovator’s dilemma: Should they construct a brand-new product using modern microservices and API-centric architecture that would require massive investments, or should they continue to invest in a cash cow slowing down the entire company’s agility?” Ghosh asks.

How to Overcome Technical Debt

Experts note the patience and long-term strategy required to overcome technical debt.

“It’s a matter of focusing on longer-term strategy over short-term financial goals,” Orlandini says. “Unfortunately for Southwest, the issues were well-known. However, the business as a whole did not have the will or motivation to invest in fixing it until it was too late. They are an extreme example but serve as a very valid case in point of what can happen if you do not understand the issues and the ultimate repercussions of not investing to avoid a meltdown, in whatever form that would take for each organization.”

Although companies want to focus their efforts on developing new products and building new features, they need to slow down the accrual of technical debt by securing old technology, says Cody Cornell, co-founder and chief strategy officer at Swimlane, which offers low-code security automation.

“Some of that capacity has to be held back to secure the product, to improve it, so people can move forward to really pay down that technical debt, and that needs to be a sustaining theme within all the work that's always going on within the organization,” Cornell says.

Companies should then determine the percentage of engineering resources to use on technical debt based on a backlog of software bugs and security issues, according to Cornell. That involves a dialogue among managers and team leads.

“Having that relationship, having an open dialogue, understanding that you can't always be moving forward, sometimes you have to work out what you have, is probably the most realistic way of solving for that and helping leadership understand what you're working on,” Cornell says.

Using automation is another way to tackle technical debt, Cornell explains.

“The beauty of software development is that a lot of it can be automated,” Cornell says. Automation can help companies overcome limited resources and improve code quality, he adds.

Paying back technical debt could be a gradual process over five years, according to SonarSource’s Gaudin. He compares fixing technical debt to plugging a leak.

“Basically, you keep changing the code, which means that if you plug the leak, and have all your developers actually deliver what we call clean code, you will actually pay back the existing technical debt,” Gaudin says. “Companies change on average about 20% of the code base. So you can imagine that after one year, you will have actually 20% of your code base, which is clean after two years. It might be like 30-35%, because there will be overlap with year one. And after five years, it can get up to 50% of the code, which is actually clean.”

IT leaders must properly communicate the pace required to develop software and not incur technical debt, according to Ghosh.

“Most importantly, CIOs need to ensure the board is patient and understands that while managing technical debt is a slow and arduous process that requires grit from the entire team, it is ultimately a strategic investment for the business that will help unlock customer value in the long run,” Ghosh says.

What to Read Next:

Reimagining the Technology Deficit

CIOs: Stop Spending on Bad Tech

CIO Lessons Learned from Southwest Airlines’ Winter Plight

COBOL, COVID-19, and Coping with Legacy Tech Debt