Row rumbles on over figures in Oracle CSO’s anti-security rant

Security researchers picking through the entrails of a withdrawn blogpost by Oracle CSO Mary Ann Davidson reckon not even her figures add up. Oracle countered that only it had access to the raw figures, so there.

Davidson's 3,000+ word diatribe against bug bounties, security researchers or customers hunting vulnerabilities in its products and reverse engineering was disavowed by Oracle. The enterprise software giant said the blog post "does not reflect our beliefs or our relationship with our customers".

Although pulled from Oracle's website, the controversial post remains available through the internet archive here.

In one key passage, Davidson claimed "we find 87 per cent of security vulnerabilities ourselves, security researchers find about 3 per cent and the rest are found by customers".

These figures – used to downplay the contribution of security researchers – have since come under scrutiny. That's perhaps because even though Oracle has disowned the whole post, the sideswipe stills smarts.

Bug hunter Adam Gowdiak analysed the Oracle CVEs to see what proportion were actually declared by researchers.

Gowdiak estimates either customers or security researchers had a role in exposing nearly 22 per cent of security flaws in Oracle's products over the last five years, a figure that is even higher in the case of Java flaws, as he explained in a post to a full disclosure security mailing list on Monday (extract below).

This compares with Davidson's figures of 13 per cent (10 per cent customers and three per cent researchers).

For Oracle CPUs released from January 2010 to July 2015, 2,210 security bugs were fixed and 485 parties credited, which yields a 21.95 per cent contribution rate.

For Java SE CPUs released from March 2010 to June 2013, 309 security bugs were fixed and 118 parties credited, which gives a 38.19 per cent contribution rate.

Assuming that credits are given to both security researchers and customers, the 13 per cent number given by Oracle CSO does not seem to be reflected by the CPU data.

Gowdiak, founder and chief exec of Security Explorations, goes on to reason that either Oracle is sitting on a large number of internally identified but still-unresolved vulnerabilities; its internal team discovered security flaws reported to it by customers or security research at or about the same time, or the "numbers provided by Oracle's CSO are bogus and have no foundation in a real-life data".

Noted Oracle bug finder David Litchfield has also expressed doubts about the same set of figures.

"If three per cent of vulns are found by external researchers and Oracle patch c. 200 bugs each quarter where are the other 6000+ CVE-IDS? ‪#science‬," Litchfield said in a Twitter update last Tuesday.

All this is, of course, based on an assessment of a repudiated blog post. In a statement, Oracle dismissed the speculation:

The explanation is simple: the majority of security bugs are found through internal testing and fixed in the normal course of development. Much of that activity is not visible outside Oracle.

In particular, many internally found and fixed security bugs are not reported through Oracle's CPU process.

Therefore, it is not possible for Adam Gowdiak or other parties outside Oracle to calculate the fraction of security bugs found and reported by customers and researchers.

Oracle has a historically poor relationship with security researchers and this has fuelled an urge to mock Davidson's disowned blog post. Security types took to Twitter to satirise the ‪Oracle‬ CSO's rant using the hash tag ‪#oraclefanfic‬, a reference to Davidson's self-declared interest in writing murder mysteries.

A more thoughtful response to the infamous Oracle blog from Katie Moussouris, chief policy officer at HackerOne – and previously the creator of the first Microsoft security bounty programmes – can be found here. ®

Bugnote

CVE stands for Common Vulnerabilities and Exposures, the industry-wide system for uniquely identifying security bugs.