Trend withdraws from 'irrelevant' VB100 anti-virus test

Trend Micro plans to withdraw its software from the Virus Bulletin 100 (VB100) tests after criticising them a poor way of deciding how well security products defend against contemporary internet threats.

Virus Bulletin's VB100 tests aims to assess how security products fare in detecting a set of viruses from the WildList, an up-to-date list of malware samples known to be in circulation. To earn the coveted VB100 certification, security software products need to reliably detect all the viruses contained in the test sample without throwing up false positives.

Trend Micro reckons that the approach might have been valid years ago when malware was written simply to spread havoc but no longer cuts any ice because it fails to assess the effectiveness of behaviour-based detection of malware or response to fast-changing threats.

Raimund Genes, Trend Micro's anti-malware CTO, described VB100 as a "20th Century test" that fails to access the "real-life" performance of security software.

"Testing is not done with an internet connection and it isn't testing for things like rootkits. Pattern matching is now only one piece of puzzle, alongside behaviour blocking technology but pattern matching is all VB100 tests," Genes explained.

Trend Micro, along with other big names in the security scene such as McAfee, has done badly in recent rounds of VB100 tests. Genes denies Trend's criticism of Virus Bulletin's testing methodology stems from these failures. "We'd be making the same criticisms even if we hadn't failed the testing. We want a more accurate test methodology," Genes said. He continued:

"AV-test, where nobody gets 100 per cent, is a better indicator of performance against malware than VB100. The standard doesn't reflect what's out there, with hundreds of thousands of new Trojans every month. Vendors have to include old signatures in databases, which slows down performance, just so they do well in VB100 tests," he added. "VB100 has great marketing value but its testing methodology is flawed."

Genes told El Reg that Trend Micro intends to pull its products from participation in Virus Bulletin tests from the second half of this year.

John Hawes, Virus Bulletin's technical consultant, defended the continuing usefulness of the tests. He questioned why Trend had first aired its gripes about the tests in public rather than do it privately.

"We've never said VB100 is the only way to test anti-malware software but it's a tried and tested approach that continues to be useful. Anti-malware products should be able to detect items of malware in circulation. VB100 is a measure of product competence and ongoing reliability that's useful for the end-users," Hawes explained.

The VB100 test approach is not written in stone. Hawes indicated a willingness to add disinfection tests and add a greater range of malware samples to its battery.

Drowning by numbers

Andreas Marx, chief exec of AV-Test.org, said that although there is nothing wrong with the actual testing performed by Virus Bulletin, the WildList sample set it tests against is hopelessly out of date.

"The threat landscape has changed dramatically, just a few years back, we had to deal with 10 to 20 virus samples per day, now we are up to 21,000 unique new samples per day, but the current April WildList only includes 678 samples - that's the number of samples we are getting on an average day in under an hour," Marx told El Reg.

"Besides this, the WildList only covers self-replicating malware such as viruses, but not today's most common threats, like Trojan Horses or rootkits. By ignoring today's reality, the list misses the really HOT samples and the numbers of samples on the WildList is too small," he added.

A changing threat landscape in the anti-malware world spurred the creation of non-profit Anti-Malware Testing Standards Organization (AMTSO). The organisation, which hopes to agree new standards for anti-malware testing. Both Trend Micro and Virus Bulletin are members of the organisation, which plans to hold its next meeting in Microsoft's Redmond campus next month. ®