Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	nick.clark	Date:	July 19, 2013 10:09:29am
Forum:	web	Subject:	Re: Why can't I access old crawled sites who have recent robot.txt exclusions?

I am wondering the same thing. Is it possible that it's just a waiting game to see a new robots.txt file? I was in the process of rebuilding a site and normally I turn off robots while I am building. How soon does the IA scan the domain to put the archive back online?