Site indexing has been the back bone of search engines since time immemorial. Search engines use indexers which crawl through websites and sore information about them inside the search engines database. Dynamic content is becoming ever more popular a design concept when implementing web sites. Traditional indexers tend to fail at indexing such sites because most of the content is generated at runtime rather than stored as separate files on the server.

Google have finally implemented a solution which might just about solve this problem. Google have opened to the public their new service which allows web administrators to issue an xml file to Google detailing their sites structure. Googlebot makes use of this file to learn how to crawl through designated website.

This solution should be very simple (and for once it actually is), but can prove to be quite problematic if the wrong approach is adopted. The trick to avoid adding extra complication to this process is to conjure up an intelligent way to generate these xml files. Google also introduced a system which allows the site administrator gzip the file containing the sitemap to reduce the amount to data that the Google will need to download.

The simplest way of writing these xml files would be to write them by hand. This would be fairly easy but hardly practical considering that small sites can sun into tens of dynamic pages and larger ones even into thousands and the person in charge of writing these files would have to adhere to the standard set by Google (which can be found here).

A more realistic way of generating sitemaps is to have a system which can automatically generate them for you. Earlier today i attempted to install a plugin written by Arne Brachhold for WordPress to add sitemap generation to this website. Needless to say the plugin is up and running and the site has already been picked up by Google Sitemaps. From the logs on the server i was able to determine that thanks to the sitemap file Googlebot was able to a magnificent job of crawling this blogger and I am looking forward to seeing the statistics that Google should be issuing to me.