Robots.txt Recommendation for WebSVN

You are using WebSVN on your server? The I recommend to have the following robots.txt in place:

User-agent: *
Disallow: /websvn/log.php
Disallow: /websvn/comp.php
Disallow: /websvn/blame.php
Disallow: /websvn/rss.php

The reason is that search engine robots could consume your server performance, especially when your Subversion repository contains larger files. But even if not: there is no reason why robots should index each and every diff between revisions. 🙂

Tags: , , ,

Leave a Reply