Search Engine Robots and Bandwidth
in WWW, Webdev / tags: design, search engines, yahoo!
Being indexed by search engines probably a good news. When I look at the statistics for this site, it seems that there are many visitors come to this site from search engines. Okey, my blog pages are indexed by search engine. I notice that search engine robots/spiders consume my hosting bandwidth. I have no problem with this since I have enough bandwidth.
You can see a sample of website statistics in the image below (the top three). I think it’s pretty big.
Yahoo!’s Slurp (search engine crawler owned by Yahoo! company) eats the most bandwidth. MSNBot and Googlebot come after that. It does not mean that Yahoo!’s search engines brings the most visitors to my site. Google does. I do not have plan of blocking those robots, even the process is so easy using robots.txt. Most search engines obeys the Robot Exclusion Standard. So, robots.txt will be just effective to block unwanted search engine robots.
This entry receives 2 comments.
kus
I have no problem with this since I have enough bandwidth
… cihhh…
Oct 19, 2007 at 5:58 pm
Thomas Arie
Kus, I was in a period when I have a very limited bandwidth. But, I think my current webhosting bandwidth is more than enough.
Well, my site traffic is not high… less than 3 Gb/mo.
So, that’s how I define the word: “no problem with bandwidth”. :)
Oct 22, 2007 at 3:23 pm