# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-Agent: * Disallow: /COP/analyzing_progress/ Disallow: /Cops/ Disallow: /docs/ Disallow: /news/ Disallow: /participant/ Disallow: /participants/ Disallow: /participants/search Disallow: /search Crawl-delay: 3 User-Agent: SEOkicks-Robot Disallow: / User-Agent: CFNetwork Disallow: /