# robots.txt for http://www.baltimoreschild.com/ # just try disallowing all bots for a while # User-agent: * # Disallow: / # just try disallowing certain folders User-agent: * Disallow /*adlaunch/* Disallow /*resources/* # this will disallow these paths as well # /bcpub/adlaunch/ # /bcpub/resources/ # didn't want to do this, but I saw a lot of log errors generated # User-agent: dotbot # Disallow: / # but I am hoping that this will override and allow Googlebot # User-agent: Googlebot # Allow: / # Disallow /adlaunch/ # Disallow /resources/ # but I am hoping that this will override and allow MSNBot # User-agent: MSNBot # Allow: / # Disallow /adlaunch/ # Disallow /resources/ # but I am hoping that this will override and allow ScoutJet # User-agent: ScoutJet # Allow: / # Disallow /adlaunch/ # Disallow /resources/ # other examples: # Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Disallow: /tmp/ # these will soon disappear # Disallow: /foo.html # Cybermapper knows where to go. # User-agent: cybermapper # Disallow: