Valid robots.txt Prevents Site From Being Indexed
So your robots.txt file looks valid, but Google Webmaster Tools says your site can’t be indexed, because it’s blocked by your robots.txt file. You’ve tried everything you can think of, but your site is still blocked from the indexing giant. Don’t panic. It’s only a matter of time!
Oops! What Not To Do.
The other day I was building a new site and figured I’d try something in WordPress that I’ve not tried before. You can’t really learn new things, if you don’t try new things.
Under the “Reading Settings” in WordPress there is an option called “Search Engine Visibility.” When you check this option WordPress will generate a simple robots.txt file for your site which includes code that will discourage robots from indexing your site. NOTE: the robots.txt file only suggests what files or directories are not to be indexed by search engine robots (also called crawlers or bots). It’s up to the actual bot to respect the request of the robots.txt file. All the major search engine crawlers will respect your robots.txt file, but beware that spammers and troublemakers most likely won’t. DO NOT use your robots.txt file to SECURE anything, ’cause it can’t. Continue reading “Google Not Updating robots.txt – Don’t Panic”