Just a little post to bait Kurt 🙂
Many people are up in arms about the idea of submitting a sample to VirusTotal and interpreting the (usually rather poor) detection count. A few links to get you started:
- virustotal usage FAIL
- why perform virustotal based av tests?
- “Only X Out of 32 Antivirus Products Detect This!”
- AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination
Now, if you read trough those, you might think that such numbers are entirely without merit and performed only by ambulance chasing amateurs who don’t know better or want go create sensationalist headlines. The truth is of course somewhere in the middle, but much of the arguments given by the anti-VT-numbers doesn’t seem bulletproof to me:
- The biggest argument seems to be that full-blown AV installations have “other means” for detecting malware which is not incorportated in the command-line scanners. Here is my problem with this argument:
- Nobody seems to be able to point the finger at what technology that should be. Is is a firewall which asks “program X wants to connect to the Internet. Allow / Deny”? Then the AV might just be perfect :-). Didier mentions McAfee’s ScriptScan, but he is the one pointing out that it is easily circumvented (and I didn’t even mention the issue that the technology is IE specific AFAIK, so all of Firefox / Opera users don’t have the extra protection.
- This is basically the same configuration which runs on e-mail gateways / http proxies, so if this doesn’t catch it, neither will your proxy!
- An other magical pixie dust feature would be “sandboxing”, which is “not used” on VirusTotal. In fact it is! Many AV products include in them an x86 execution engine and use it for unpacking or detecting different behavior. Of course this is limited by time, but it is still “executing in an emulated environment”. (Two products which rely heavily on this and are present on VT are Norman and ESET NOD32) – so you have your sandboxing
The gist of the matter is: if it can’t detect it “offline”, you’re in very muddy waters, praying that the “online” detection will catch it before it can do real damage (for example it is conceivable that such online detection works on the basis of accumulating scores, which would mean that the file has done several dubious actions before it accumulated a high enough score) and then there is the whole issue of bugs in these software.
- Virus signatures are not updated frequently – well they are updated more frequently than the ones on the average user’s computer.
- AV software is not configured properly – the guys at VT are very good about getting back to you and I’m sure that if such a concern would be a real one, companies would have already advised them on how to do it.
- The fact that no detection is returned might mean that the AV timed out – so what? Most desktop AV programs contain hard limits on execution time and file size (to avoid making the computer unusable) and if either of those is tripped, the file is not scanned!
So there you have it: the detection in the “real world” might be slightly better than on VT, but not so much better that you can disqualify the results.