Automatically notify Google and MSN when your site is updated

This is a migrated post from my former blog at bloodsweatand.net. It was originally posted 2009-03-22.

I recently wrote about XML sitemaps and how to generate them with a ASP.NET handler. Once you have a XML sitemap you can notify search engines such as Google and MSN about it in three ways. You can direct their crawlers to it with a robots.txt file, submit it to them using their webmaster tools and you can ping them. The two latter gives you the option to notify the search engines each time your website, and thereby your sitemap, is updated. Doing so using webmaster tools is easy but requires manual work and that the editor who publishes content on your site knows that it should be done and how to do it. So, the third alternative, pinging Google and MSN, which are the search engines that I know of that so far offers this ability, looks like a pretty good alternative.

One way to implement this functionality would be to ping the search engines each time a new page, blog entry etc is created. Google does however recommend that it shouldn't be pinged more than once per hour. Therefore it might be a good idea to create a scheduled job that checks if the sitemap has been updated once per hour, and if so pings the search engines.

I've put together a small class library that is able to handle both alternatives. You can download the source code, including a console application for testing, here as a zip file.

The UpdatedSitemapNotify method

The class library contains a single public class, SitemapUpdateNotifier. SitemapUpdateNotifier in turn only contains a single public method which has two overloads, UpdatedSitemapNotify() and UpdatedSitemapNotify(DateTime? updatedAfter). The overload with no parameter will ping the configured targets, no questions asked.

The overload that takes a nullable DateTime object as a parameter will first check if the sitemap, or sitemaps in case the configured URL points to a sitemap index, have been updated after the date specified in the parameter and will only ping the targets if the sitemap have been updated after the specified date. If the parameter is null it will not perform this check and assume that all sitemaps should be used when pinging.

The targets that the method will attempt to ping are also read from the applicaitons configuration.

Configuration

In order to use the SitemapUpdateNotification class library you will have to add a new configuration section to your web.config.

<configuration>
  <configSections>
    <section name="sitemapUpdateNotification" 
      type="SitemapUpdateNotification.Configuration.SitemapUpdateNotificationConfiguration,
      SitemapUpdateNotification" />
  </configSections>
</configuration>

The config section, placed directly under the configuration node, should look like this if you intend to ping Google and MSN:

<sitemapUpdateNotification 
  sitemapUrl="http://mydomain.com/Sitemap.ashx">
  <notificationTargets>
    <add name="Google" targetUrl="http://www.google.com/webmasters/tools/ping?sitemap=" />
    <add name="MSN" targetUrl="http://webmaster.live.com/ping.aspx?siteMap=" />
  </notificationTargets>
</sitemapUpdateNotification>

A note about the solution

It may seem a little odd that the webserver fetches the sitemap from itself using HTTP. The reason I choose this solution was to keep it as general as possible and not to create a second place for the logic for determining when a page has been updated.

PS. For updates about new posts, sites I find useful and the occasional rant you can follow me on Twitter. You are also most welcome to subscribe to the RSS-feed.