Automatically notify Google and MSN when your site's content is updated.
I recently wrote about XML sitemaps and how to generate them with a ASP.NET handler. Once you have a XML sitemap you can notify search engines such as Google and MSN about it in three ways. You can direct their crawlers to it with a robots.txt file, submit it to them using their webmaster tools and you can ping them. The two latter gives you the option to notify the search engines each time your website, and thereby your sitemap, is updated. Doing so using webmaster tools is easy but requires manual work and that the editor who publishes content on your site knows that it should be done and how to do it. So, the third alternative, pinging Google and MSN, which are the search engines that I know of that so far offers this ability, looks like a pretty good alternative.
One way to implement this functionality would be to ping the search engines each time a new page, blog entry etc is created. Google does however recommend that it shouldn't be pinged more than once per hour. Therefore it might be a good idea to create a scheduled job that checks if the sitemap has been updated once per hour, and if so pings the search engines.
I've put together a small class library that is able to handle both alternatives. You can download the source code, including a console application for testing, here as a zip file.
The UpdatedSitemapNotify method
The class library contains a single public class, SitemapUpdateNotifier. SitemapUpdateNotifier in turn only contains a single public method which has two overloads, UpdatedSitemapNotify() and UpdatedSitemapNotify(DateTime? updatedAfter). The overload with no parameter will ping the configured targets, no questions asked.
The overload that takes a nullable DateTime object as a parameter will first check if the sitemap, or sitemaps in case the configured URL points to a sitemap index, have been updated after the date specified in the parameter and will only ping the targets if the sitemap have been updated after the specified date. If the parameter is null it will not perform this check and assume that all sitemaps should be used when pinging.
The targets that the method will attempt to ping are also read from the applicaitons configuration.
In order to use the SitemapUpdateNotification class library you will have to add a new configuration section to your web.config.
<configuration> <configSections> <section name="sitemapUpdateNotification" type="SitemapUpdateNotification.Configuration.SitemapUpdateNotificationConfiguration, SitemapUpdateNotification" /> </configSections> </configuration>
The config section, placed directly under the configuration node, should look like this if you intend to ping Google and MSN:
<sitemapUpdateNotification sitemapUrl="http://mydomain.com/Sitemap.ashx"> <notificationTargets> <add name="Google" targetUrl="http://www.google.com/webmasters/tools/ping?sitemap=" /> <add name="MSN" targetUrl="http://webmaster.live.com/ping.aspx?siteMap=" /> </notificationTargets> </sitemapUpdateNotification>
A note about the solution
It may seem a little odd that the webserver fetches the sitemap from itself using HTTP. The reason I choose this solution was to keep it as general as possible and not to create a second place for the logic for determining when a page has been updated.
- XML sitemap with ASP.NET
- Creating a custom Configuration Section in .NET
- XML sitemap with ASP.NET MVC
- Template for RESTish JSON service with WCF
- Extending ASP.NET MVC Music Store with elasticsearch
- Handling multiple ASP.NET configuration files with build events
- Twitter style paging with ASP.NET MVC and jQuery
- Performing and Testing Redirects with ASP.NET Web Forms MVP