What 108M Lines of Code Tell Us
Results of the first annual report on application quality have just been released by CAST. The company analyzed 108M lines of code in 288 applications from 75 companies in various industries. In addition to the ‘usual suspects’ – COBOL, C/C++, Java, .NET – CAST included Oracle 4GL and ABAP in the report.
The CAST report is quite important in shedding light on the code itself. As explained in various posts in this blog, this transition from the process to its output is of paramount importance. Proficiency in the software process is a bit allusive. The ‘proof of the pudding’ is in the output of the software process. The ability to measure code quality enables effective governance of the software process. Moreover, Statistical Process Control methods can be applied to samples of technical debt readings. Such application is most helpful in striking a good balance in ‘stopping the line’ – neither too frequently nor too rarely.
According to CAST’s report, the average technical debt per line of code across all application is $2.82. This figure, depressing that it might be, is reasonably consistent with quick eyeballing of Nemo. The figure is somewhat lower than the average technical debt figure reported recently by Cutter for a sample of the Cassandra code. (The difference is probably attributable to the differences in sample sizes between the two studies). What the data means is that the average business application in the CAST study is saddled with over $1M in technical debt!
An intriguing finding in the CAST report is the impact of size on the
quality of COBOL applications. This finding is demonstrated in Figure
1. It has been quite a while since I last saw such a dramatic
demonstration of the correlation between size and quality (again, for
COBOL applications in the CAST study).
Source: First Annual CAST Worldwide Application Software Quality Study – 2010
One other intriguing findings in the CAST study is that “application in government sector show poor changeability.” CAST hypothesizes that the poor changeability might be due to higher level of outsourcing in the government sector compared to the private sector. As pointed out by Amy Thorne in a recent comment posted in The Agile Executive, it might also be attributable to the incentive system:
… since external developers often don’t maintain the code they write, they don’t have incentives to write code that is low in technical debt…
Congratulations to Vincent Delaroche, Dr. Bill Curtis, Lev Lesokhin and the rest of the CAST team. We as an industry need more studies like this!
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)
Comments
Anthony Mak replied on Tue, 2010/09/28 - 9:07pm
Hi Israel,
Great article.
>… since external developers often don’t maintain the code they write, they don’t have incentives to write code that is low in technical debt…
I find your reason why government code seems to have poorer quality very englightening.
So as Size increase -> Total Quality Index decrease. Is there any study or paper on how best to tackle or reverse this problem?
In your study, are there any special circumstances where increase in Size do not lead to a decrease in Total Quality Index?
Anthony Mak
Alessandro Santini replied on Wed, 2010/09/29 - 6:46am
Dear Dr. Gat,
thanks for this interesting reading - I am relatively new to the concept of Technical Debt and statistics are indeed not my best area of expertise but nevertheless I tried to draw some conclusions out of this article.
This makes me wonder if there is uniformity in the kind of samples adopted; are all these applications written for the same industry? Is their logical complexity comparable? Have these application been developed by teams with comparable skillsets, experience and education?
Thanks.
Alessandro
Stephane Vaucher replied on Thu, 2010/09/30 - 1:01pm
Baruch Atta replied on Mon, 2010/10/25 - 3:32pm
In other words, it is all still a hypothesis, black magic, waving of wands, wearing of funny masks.
We have known for decades what good code looks like. We know what bad code looks like. We know if a program can be modified easily. We know that good systems have good documentation.
So, I view with much skepticsm any new attempt to measure "application quality".
As for "average technical debt per line of code" measures, I have no doubt that bad, spaghetti, undocumented code is more costly to maintain, and the figure of $2.82 per line of code may be accurate. Or not. What is that figure compared to? What is the average technical debt per line of code for well written, well documented code?
The difference between the cost to maintain bad code to the cost to maintain good code, that is important. Is the cost of extra maintenance of bad code worth the re-engineering effort to transform to good code? That is the "to be or not to be" question. Is it worth it to re-engineer? I say - in every case - YES!
Is it better to migrate to a "more modern" platform, from COBOL to JAVA?
I say, in very many cases, no. No, because the risks of migration are high, and the costs are even higher. The return is marginally small.
It is probably cheaper and less risky to migrate from a bad coded COBOL system to a good coded COBOL system, and then if desired, to a new platform, than it is to migrate directly to a new platform. I know, I have done it a few times.
Two final comments.
Performing a GOTO to a paragraph-end is considered good programming practice in COBOL. It is an easy way to bypass without deep nested IF statements. And it doesn't add to complexity, just the opposite.
And I would disagree with the report, and say that the reason that government owned systems are harder to maintain is not because of the outsourcing. The real reason is due to strict cost controls, and the lack of constant code improvement efforts. That is, government code is left to run longer without modifications, than commercial systems code. It is the cost containment in government systems that costs more in the long run. Penny smart but pound foolish. That's my two cents.
Baruch Atta