I share your concerns about testers who don’t have analytical skills (and other relevant skills), and have been trying to raise interest in some form of realistic certification for testers for some time (Andrew Lee and I presented at Virus Bulletin on that topic this year.)

There is further documentation in preparation about malware creation for testing purposes. That’s a complex issue, and I think the industry hasn’t done itself favours by concentrating on the ethical and safety issues rather than highlighting the technical problems.

Statistical relevance is certainly an issue, but so is statistical validity. There are many instances where incorrect conclusions have been drawn from the data. I’m hoping someone will pick up the gauntlet on documenting stats and detection testing, sooner or later.

Unfortunately, dynamic testing is always likely to attract smaller sample sets, because of the complex and resource-intensive nature of the methodology. It was probably not a good idea to give the impression that a 50 sample test set is generally sufficient, but what constitutes a valid test set is very context-specific. If you were testing Windows CE virus detection for example, that would be too large a sample set… (Yes, that’s an extreme example!) By the way, your figure for monthly samples is way, way too low, though exact numbers depend on how you measure.

Anyway, you raise some very interesting points, and I’ll probably look at them in more detail on the ESET blog in the near future.