If you are a webmaster, I am quite sure that you have faced before Search Engine bots. They are little programs sent by Search Engines like Google or Yahoo! to spider or to index your website in order for your webpages to appear in their Search Engine results. They are very important, especially nowadays when we are flooded with new websites everyday. You need to get visitors from Search Engines. They are one of the most important traffic generators for most websites on the Internet. However, not all bots are good. Some bots are sent by email harvesters or some unknown Search Engine whose purpose is just to index as many sites as possible in the shortest period of time. The problem with that is it may crash or slow down your server and thus preventing others from accessing your site. Worse are those bots which are improperly coded which may result in endless indexing of your site. This again creates problems for you and it certainly doesn’t benefit you (I doubt that you will get any traffic from such Search Engines), even if you have a lot of bandwidth and server resources. They are also bots which ignore normal spider “etiquette” and do their own stuff like indexing hidden files or follow links which are not meant to be followed. Don’t you just hate irresponsible bots? They ought to be shot.

The sad truth is that a lot of webmasters don’t know that they have been attacked by “bad” bots. I must admit I used to be one of those who thinks that ignorance is bliss. However, you will be surprised that one day, this ignorance can really hurt you when your server goes down or that your webhost refuse to host your site on their servers because of these bots hammering on their servers. You will have to spend time to search for another webhost, upload your files again and wait for DNS propagation. Big big waste of time and your site will not be accessible during this time. And it will just happen again as bots keep on attacking you. These bots will never stop unless you do something about it.

My advise to all of you webmasters is to do some basic protection early. Check your website logs and see which bots (and their IP addresses) have been tracking you and try to block them. These bad bots usually attempt to navigate the easiest prey, so you should make it as tough as possible for them to get through. If you need more information on how to block these bots, take a look here. It’s a very good site who a lot of information on how to spot bad bots and how to block them. Most of the solutions provided are easy to follow.