Disclaimer: these are my own opinions and they do not necessarily reflect the opinions or policies of any of my current or past employers.

There is a class of applications which can be categorized as greyware: programs which can be used for both good and evil. A few examples (in no particular order):

• nmap, the most famous network scanner: it can be used during a security assessment or it can be used by an attacker to search for possible attack targets.
• VNC servers, and other remote control software (RAdmin for example): both legitimate users and illegitimate users can install / use these software.
• FTP servers: can be used to share files knowingly or unknowingly (if installed stealthily by an attacker)
• Password enumeration tools: can be used for forensics for example, or to steal passwords.

These few examples already demonstrate that there are many shades of gray out there – so to speak. Philosophically you can argue that software isn’t good or bad and that it becomes one way or the other during its use. However AV software isn’t HIPS, it is intended to give a verdict before the particular piece of code runs. As such during analysis the virus researcher must decide if there is a reasonable possibility that this piece of code can be used in a malicious manner (of course if it’s clearly malware, it will be signed as such).

To give you an example how such an attack would work, imagine the following (very real) example: a set of commercial tools (probably cracked version of them) packaged together in a self-extract archive which contains an installation script. Many archiving products offer the possibility to have some kind of script automatically executed after extraction and there is malware out there which uses installation products directly (like NSIS or Inno Installer). The package includes a commercial keylogger preconfigured to send the capture keystrokes to an e-mail address under the attackers control and a commercial FTP server set up to share the C drive for example with a preset password on a preset port.

One can argue that all of the code used here is not malware (even the installation script included in the self-extract archive can be part of a one-step install package put together for some particular – benign – purpose). The fact that this package is malware can be only determined either by examining the source of it (if it was spammed out to a million people with a misleading message or downloaded by using a browser exploit) or the context of the usage (ie. asking the client if s/he knows about these programs and if s/he agrees to them running).

This situation has no clear solution: only detecting the package provides no proactive protection and is mostly useless (because many times it is set do delete itself after execution and what you want to detect are the dropped programs) and detecting the particular programs (this sentence seem to have a lot of p’s in it :-)) means that legitimate users who knowingly want to use these programs will be annoyed by the detection.

The current trend in the industry is to err on the safe side. Applications with missuse potential will usually be put in a separate category (like Possibly Unwanted Programs) and the prompt displayed by the realtime scanner will differ in the description (possibly also in the design) and in the offered options (which in a clear malware case are something like delete / quarantine and in such cases they are something along the lines of allow / deny / always allow).

On the flipside many users are annoyed by these prompts (in a good case) or don’t read the text of the prompt at all and assume that the software is infected by a virus!. This can cause headaches and additional support costs / lost revenues for the software vendor, which usually blames the AV software as the first response.

While I don’t know any silver bullet solution to this problem, here are some tips on how to avoid making your software look dubious:

• Don’t use an executable packer! They offer minimal protection against crackers, the people who are using an illegitimate version of your software most probably wouldn’t use it if it wasn’t free for them and the more agressive packers (like the ones detecting Virtual Machines and/or debuggers) makes your program look dubious. Not to mention that some packers insist on loading a driver for unpacking purposes, which: (a) makes the clients system potentially less stable – in kernel mode one wrong move and you brought down the whole system (b) limits the use of your software – people running with limited privileges (which is a good thing) or on non-standard hardware (like x64 systems) might not be able to use it and (c) reduces the performance of your application.
• Along the same lines: don’t try to detect Virtual Machines and/or debuggers! Most of these detections can be easily overcome, but they can lead to believe anyone looking at your software that it has some hidden purpose and/or doesn’t want to be analyzed (Virtual Machines are often used during analysis because they serve as a sandbox environment and can be easily bought back to their initial state).
• Don’t put executables / libraries in the windows / system directory unless you absolutely need to! Your program files have a very well defined place: the program files directory!
• Follow conventions! Use the documented ways to register at start-up and use descriptive names for your registry entries!
• Create a version info for all of your executables / libraries, preferably with a text that includes a link to a website where the purpose of the program is explained!
• Don’t use encrypted strings in your application! They can be easily broken (since the idea is that your application needs to decrypt them at some point, meaning that anyone looking at your application can simply run the decryption routine included in it) and make it seem like your application needs to hide something!
• Don’t overdo protection of your program! Example: some chinese messenger applications use drivers to protect themselves. Ignoring the fact that you cut off some part of your users (who run as limited user or use a possibly unsupported platform – like Vista or WinXP x64) and endanger the stability of their system, you make your program look fishy, especially in this day and age where almost every semi-advanced piece of malware tries to include rootkit technology.
• If your program works from the command line, make it such that when ran without any parameter, it displays the help, which should include a link to your site. This gives anyone looking at the program a good idea what it is about.
• Don’t add any functionality which makes it easy to automate (like running it from the command line). Use a GUI interface. While this might be hard to do sometimes (because many of your more advanced client expect to be able to script the functionality of the program, consider it. Not providing an easy way to automate your software makes it a less desirable target to be included in malware bundles.
• Update: avoid using the services of individuals / companies who are known to host malicious files / send out spam willingly. Two examples would be: the estdomains registrar and Chinese / Russian / Romanian hosts. Disclaimer: I know that this is a sweeping generalization and that in all of these countries the number of legitimate businesses far outweigh the illegitimate ones. Also, just recently a report came out identifying the US as the primary source of attacks / malware / SPAM. The perception is still there however and most probably a little money saved can cost you a lot in sales later. Hopefully we can clean up our act and gradually remove these stereotypes.

One thing I must say is that even after following these guidelines the software might end up in the special category I talked about earlier if it has a miss-use potential and probably has already been miss-used. In this cases you don’t have a magic solution, because they are not misrepresenting your product, the customer is not reading the warning. The one thing you can do (besides explaining the situation to your client) is to give them instructions on how to except the given executable from scanning (most AV products include options to except certain files / directories from on-access verification) – by downloading a trial version of the scanner and installing it on a test machine or a VM (Virtual Machine). You could of course advise them to contact the AV company for support, but probably this would considerably reduce your clients satisfaction.

Of course if an AV clearly mis-categorizes your product (for example it says that it is a trojan or worm), most probably it is a honest mistake or a generic detection gone bad. In these cases you can most likely get the vendor to fix the problem in a short time, because it helps them to improve their customers satisfaction.

Update: check out the post on the Authenium blog about the same topic.