Search engine spiders - eeeeew

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

What if i would tell you, that you have spiders on your website. I think most of females would go “eeeeew” if they heard about a big hairy spider. But in this case i’m talking about digitalized spiders that are on your site almost each day and have huge appetite for… data. Yes they feed on data from your sites or blogs. They live from content and updates you make and you really want to give them food, lot’s of it.

These spiders are a software program, that is crawling your sites regularly. They gather all the data from your sites and add them to the search engine database where algorithms are applied and the data is being analyzed. At the end, you find your site or web page in the search engine index which is shown as a SERP or Search engine result page where you will find your listing.

So webmasters really adore these cute little cuddly things called search engine spiders. They… erm… We depend on them and that is the reason why we also need to add content daily, the more we feed them with fresh and unique content, the more they will like us and come back for more. This means our pages will get into the Google or other search engines faster.

Also we need to know what kind of food these spiders like. They like fresh and unique content, daily or even 2 or free times a day. They like css and they like clean sites that are xHTML validated, meaning they can read/eat it without problems. The list about what they don’t like is a bigger one, they don’t like big and long pages, they only spider about 10kb (excluding pictures, videos and that kind of stuff) of a page so you should have smaller pages in large numbers. Pages that have 50-100kb or more spiders does not like. They also don’t like a lot of coding or javascript so use this externally, same with css. They also don’t like tables,pop-ups, frames, etc. So keep it clean!

How to know when a spider crawls your site? If you have server side statistics or a counter, you should notice names like MSNBot, Inktomi Slurp, Googlebot, EchO!, AskJeeves, Alexa (IA Archiver) and others. These are the spiders you really like to have daily.