How to create a well behaved bot?

From the ever watchful master of bots and scrapers incrediBILL (:-)) comes the following tip, which you might find useful if you yourself plan / need to implement a web crawler as part of your product / service.

, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *