Repeatable Security Analysis: A Research Challenge

Have you ever used a security analysis tool on your code?


Those who have tell me consistently the problem: they get lots of ‘false positives’ – error warnings that turn out not to be security problems in practice. What, however, can we do about that?

Now the false positives are not in themselves an insurmountable problem: a developer will typically read through the list of errors, and ‘triage’ them into ‘errors I need to fix’; ‘not errors at all’; and ‘errors that I don’t plan to fix’. Once the ‘need to fix’ errors have been sorted, the remainder can be ignored. The problem is what happens next time the tool runs. The developer must do the whole job again. 


Indeed, most experts would advocate putting secure code analysis into the build chain, with a policy of no warnings for the released code. That is hardly consistent with a security tool that throws up hundreds of ‘false positive’ messages.


What we want is for the tool to record and remember the triage decisions, to filter out the false positives. This is, as far as I know, impossible with the current generation of tools. Indeed, when you think about it, this presents quite a hard problem:


  • It requires a persistent record of which errors have been triaged out, and
  • Changes to the code may invalidate the original triage, requiring the programmer to review the decisions again.


The persistent record presents a significant implementation challenge. Effectively it is an artefact, representing programmer effort, so it belongs the source code control system. It might be implemented as annotations to the code, but there’s an obvious concern that it may make the code difficult to read and understand. The alternatives, of putting the information in one or more separate files, also present problems, since the files must also go into source code control.


Spotting the impact of code changes is also a hard problem. Security errors are often, if not mostly, detected by tracing through code, and so typically depend on a large number of lines of code. How do we decide when one of these lines of code has changed? The answer can only be through integration with the source code control system. 


And finally, assuming a tool implements both of these features helpfully, we then have a further issue: it is certain to require significant processing and disk access. How do we make the tool run fast enough to produce practical feedback for a developer?


All those challenges make this a difficult problem to solve. Until we do solve it, however, even the best security analysis tools will be virtually useless to programmers in practice. 


Let’s solve it, soon!


- Charles