software cost estimating
That article does not understand software economics. COCOMO is used as a very rough estimate for early development, when most of the work is cookie-cutter code. It is also generally disputed and not viewed as accurate for projecting real effort. For that matter, LOC is not a particularly good metric for assessing a code base: not for assessing concepts of "maintainability" or "cost" or "defect rate" or any other metric of code health. It is often used as a base starting point (i.e., given everything else being identical, a code base 2x as large will cost roughly 2x as much), but all the other factors are often considered more important. "All the other factors" include things like modularity, code complexity (many methods of measurement), token count (similar to line count, but not just measuring lines; i.e., some lines have a LOT of tokens, while other lines have only one token), and more.
I used to work for a company that had many patents around concepts of code complexity. We were one of many companies with a "cost estimator" tool, too. And we all knew (and openly admitted) that our estimator (just like all others) was very, very, very rough. I.e., cost could easily be 50% of our estimate or 2x our estimate -- the statistical deviation was wide. Our statistical relevance was so-so at best. Same with other models. Finance people still loved the models because you have to assess cost projections somehow, even when you know the estimates are widely flawed.
Side note: this is one of the many "holy grail" targets. If you can make an accurate (good luck) cost projection tool to help organizations identify "cost to complete a software project," then you have a product you can sell for $$. Just don't step into the ring assuming that nobody has tried. There are literally dozens of companies shilling various models, and they all sorta suck. Very wide margins-of-error (statistical deviations). As of a couple years ago, there was not a product in existence that is substantially more accurate than quality engineering managers giving a "guestimate" based on decades of experience. Some companies sell massive reports that combine all of the current estimation methods (we sold our estimates to a couple prominent report-generation companies, like Black Duck), but there is no single "this number is pretty good" solution. At least not as of ~2 years ago; last time I was working in that field.
60 Replies
thank you
I am well aware that people will come in here saying:
Well my org uses ABC-XYZ Method, and it's been great for us.Super cool. I am still confident that ABC-XYZ Method, when used on tens of thousands of code bases, will come up as "meh" at best. I say that because we ran all of the methods with open math against 16,000 code bases, and saw widely varying results. And we had the real cost results (from clients) for ~1k code bases. None of the models were consistently accurate.
So Would you have a guesstimate for how much LLVM cost was ?
I guess one closing comment:
Black Duck's "just get all the methods into one report" was probably the best. It just gives Merger & Acquisitions (M&A) teams a TON of data to consider. The M&A teams can decide what they feel is the best cost estimates for the software they're acquiring.
But really: get some senior managers who've been through stuff and can give "A Grizzled Vet's Estimate."
oh hell no š
Way out of my expertise. I was part of the team that was building the estimation software, but like we were implementing the math from "white paper" into "code." And scraping for code bases. And running tests. Doesn't mean I know the stuff; I couldn't fully invent or even explain the best actual software estimation models, let alone try to assess cost on a project. š
But Chris might have perspective on real costs. Maybe the team kept time & resource logs -- both human and infrastructure. I don't know.
I guess given the wide deviations we saw in our lab experiments, and that COCOMO estimate of $500m, I'd hazard a wide-ranging guess of:
Anywhere from $250m (1/2) to $1b (2x).š my cheap cop-out answer oh, another HUGE complicating factor:
How bullet-proof must this software be?I.e., building 100k-LOC video game is much cheaper per LOC than building a 100k-LOC military aircraft controller -- lives are at stake, the software is classified, testing is rigorous, government regulations must be followed, etc, etc, etc, etc.
The GHC estimate is forgetting almost all of those devs are PhDs who are doing the development on grant funding.
This makes me realize how valuable LLVM was, and at the same time how cheap it is given that it runs almost all silicon on earth.
I think the article was playing the "what if Microsoft had made it internally" type of analysis game.
Have you seen some of the stuff in GHC?
That was almost an internet meme in the 90s, like with Linux and the rise of FOSS.
There are haskell features from the 90s which were considered new and confusing when Scala tried to introduce them to a general audience decades later.
I don't recall the acronym "GHC" off-hand.
Haskell
oh! The Haskell thing. Yeah. I saw that one.
It's the haskell compiler.
I don't know much about it.
but I know it has some cost and complexity estimators included, right?