Pages blocked in robots.txt can’t be indexed (19/35)

List item

No Title

No Description


Google generally honors the request not to actually crawl these pages, but if they’re linked to FROM crawlable pages, they’ll sometimes be indexed even though their actual content was not viewed. The description in the searching listing will generally say A description for this result is not available because of this site’s robots.txt when this is the case. Use a meta robots HTML tag set to noindex if you don’t want a page indexed. 

Contributors: Mark Brown from The Zebra

Written by Zak Parker

Journalist, writer, musician, professional procrastinator. I'll add more here later.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.