From the ever watchful master of bots and scrapers incrediBILL (:-)) comes the following tip, which you might find useful if you yourself plan / need to implement a web crawler as part of your product / service.