######################################################### # -- NETPOINTERS INTL. -- # # http://www.netpointers.com # # # # Vestergade 18 B # # DK-1456 Copenhagen K # # # # Robot exclusion file. Sets recommended access rights # # for known and unknown robots. # # # # Created: 2001-07-27 # # Author: Rasmus T. Mohr (rmo@netpointers.com) # # Modified: 2004-03-17 by Jan Schenling # # # # (c) Netpointers Intl. 2001 # ######################################################### # -- Known bots -- # # AltaVista -> av User-agent: Scooter Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # DirectHit User-agent: Grabber->dh Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # Excite -> ex User-agent: ArchitextSpider Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # Fast -> fs User-agent: Fast Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # Google -> go User-agent: Googlebot Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # Inktomi -> in User-agent: Slurp Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # MSNsearch -> ms User-agent: msnbot Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # -- Unknown bots (i.e. all the others) -- # User-agent: * Disallow: Disallow: /asp-filer/ Disallow: /bilder/ # -- Email collectors, density analysers --# User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: NetMechanic Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Keyword Density/0.9 Disallow: /