# only allow website/index.asp to be crawled by spiders User-agent: *