Ebusiness4all Blog
Discussions on the evolving online world
November 29th, 2005

Helping the robots

Spiders look for the robots.txt file in your web site home directory to tell them what they can index.

Have you checked your site robots.txt file, is it allowing the spiders in or blocking them?

The file is a text file with a simple structure such as: -

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-images/
Disallow: /wp-includes/

This file allows in all the spiders but prevents them from visiting the named sub-directories.

Why don’t you do a search in Google for Robots.txt Validator to find the many online services which will report the content of your robots.txt file and let the search engines into your site.

November 17th, 2005

XHTML compliant code

I met an SEO consultant the other day and he stressed the importance of XHTML code for search engine spiders.

He stated that 80% of sites had bad HTML and spiders staooped spidering when they met an error.

This got me thinking – how many commercial sites have bad code and how much value / opportunities are they missing.

Do you know of any top brand names with poor site code?