Static Analysis & Vulnerability Rates
After a long hiatus, I’m back at my blog again. Sorry for the absence, but there were a number of reasons, many out of my control. Nevertheless, I only try to bother other people when there’s something interesting to talk about. I’m sure none of you are lamenting the fact that you have a few extra hours available for other activities. At least I hope not.
Today, I’m going to talk about a report I recently read from Coverity. It’s called the Coverity Open Source Integrity Report 2010. Each year they scan open source software for vulnerabilities and publish the results. This year they included Google’s Android operating system in their scan.
There are a couple of things that I liked about the concept and process behind this report. First, I have to be clear about one thing. Showing hits from a scan tool does not mean that vulnerabilities exist. Think of this as a patient showing symptoms of a disease. It takes additional diagnosis to verify that the disease does exist. However, with that knowledge, I think it’s helpful for the industry to have someone scanning large software projects for potential vulnerabilities. I’m certain that it helps Coverity improve the quality of their static analysis tool. At the same time, publishing the results helps us establish baseline metrics for other projects.
A few years ago, I worked with an engineer who was asked to assess a large internal software project for a customer. As part of the project, he benchmarked the software’s static analysis score against two similar sized projects in the open source community. This turned out to be a valuable benchmarking mechanism and simplified the decision on what to fix. In many ways, we humans are better at making comparative analyses instead of independent assessments without a comparison. If you want to test this theory, go out into an open field and throw a ball as far as you can. How far did you throw it? Now perform the same test on a football field. It will be much easier to estimate the distance.
Back to the report. They compared Android to conventional open source software, saying that Open Source has an average vulnerability rate of 1 vulnerability per 1000 source lines of code (KSLOC). This only represents security bugs, so it sounds reasonable. Android’s average was 0.47 per KSLOC.
Now, if I were writing a book, I’d get into the vagaries of the measurement process and what percentage of those vulnerabilities might actually be exploitable. But this is a blog post, so I’ll get to the point. If you are just getting started in the business of software security and need to talk to management, this kind of thing can help. Starting with a static analysis tool and having a benchmark, you can begin to measure your own software and figure out how it compares to open source. If you use a lot of open source software in your projects, you can probably estimate the amount of vulnerabilities introduced by that software and get a rough measure of the risk. If the security of your software turns out to be much worse than software built by volunteers, tell that to management. If it comes out at release showing less than 1 vulnerability per KSLOC, say that too, but also look at the severity. If nothing else, proposing this kind of testing may be enough justification to purchase a copy of a good static analysis tool. Then you will at least have a rough idea of the size of the software security job in front of you.
Having the work of years boiled down into a single statistic that you can use with senior managers will make it easier to manage up in your organization. Managers like simple baseline measures and rules of thumb. Once you have management support for your program, then the real work begins. Good luck.
Jim Molini, CISSP, CSSLP
Let me start by saying that I was 