Google's Internal Debate on Noindex in Robots.txt

John Mueller, from Google, recently shared his thoughts on using the noindex directive in the robots.txt file of his personal website. He talked about the benefits and drawbacks of search engines supporting this directive and provided some insights into Google's internal conversations on the topic.

Mueller’s robots.txt has sparked discussions over the past week due to the unusual and non-standard directives he included in it.

It was only a matter of time before Mueller’s robots.txt file caught the attention of the search marketing community and started circulating widely.

Noindex Directive

Everything that’s in a robots.txt is called a directive. A directive is a request to a web crawler that it is obligated to obey (if it obeys robots.txt directives).

There are guidelines for writing a robots.txt directive. Anything that does not follow these guidelines may be overlooked. In Mueller's robots.txt, a non-standard directive caught the attention of someone. This person decided to ask John Mueller on LinkedIn if Google actually supported this non-standard directive.

This is a valid question because it's common to think that if a Google employee is using it, then Google must support it.

The non-standard directive used was noindex, which is a part of the meta robots standard but not the robots.txt standard. Mueller did not just use the noindex directive once, he actually had 5,506 noindex directives.

The SEO specialist who asked the question, Mahek Giri, mentioned this in their inquiry.

“In John Mueller’s robots.txt file,

there’s an unusual command:

“noindex:”

This command isn’t part of the standard robots.txt format,

So do you think it will have any impact on how search engine indexes his pages?

John Mueller curious to know about noindex: in robots.txt”

Why Noindex Directive In Robots.txt Is Unsupported By Google

Google’s John Mueller answered that it was unsupported.

Mueller answered:

“This is an unsupported directive, it doesn’t do anything.”

Mueller explained that Google had once thought about supporting the noindex directive in robots.txt. This would allow publishers to prevent Google from crawling and indexing their content simultaneously.

Currently, you can either block crawling in robots.txt or block indexing using the meta robots noindex directive. However, it is not possible to block indexing with the meta robots directive and block crawling in robots.txt at the same time. This is because blocking the crawl will prevent the crawler from detecting the meta robots directive.

Mueller explained why Google decided to not move ahead with the idea of honoring the noindex directive within the robots.txt.

He wrote:

There were debates about whether it should be included in the robots.txt standard. The idea was to allow blocking both crawling and indexing simultaneously. Currently, robots.txt allows blocking either crawling or indexing (if crawling is allowed, you can use a robots meta tag to block indexing). The proposal was to enable a "noindex" directive in robots.txt to block both actions.

However, it was determined that this feature should not be supported as part of the robots.txt standard. This decision was made over a decade ago due to the risk of people unintentionally deleting crucial parts of their website when copying and pasting robots.txt files without thorough inspection.

Why Did Mueller Include a Noindex in Robots.txt?

Mueller explained that Google is unlikely to support the use of the noindex tag, which was confirmed about ten years ago. The revelation of these internal discussions is intriguing, but it also adds to the mystery surrounding Mueller's robots.txt file.

See also: 8 Common Robots.txt Issues And How To Fix Them

Featured Image by Shutterstock/Kues

Editor's P/S:

John Mueller's use of the noindex directive in his robots.txt file highlights an ongoing debate within Google over the potential benefits and drawbacks of supporting this feature. While Google initially considered implementing the directive to allow publishers to simultaneously prevent crawling and indexing, concerns over unintentional website deletions led to the decision not to support it.

Mueller's unusual directive has sparked further discussion among the SEO community, raising questions about the reasons behind his decision to use it despite its unsupported status. Mueller's explanation that the directive is "an unsupported directive, it doesn’t do anything" suggests that he may be using it for experimental purposes or as a placeholder for future implementation. However, the true purpose remains unclear, adding to the intrigue surrounding Mueller's robots.txt file and the ongoing evolution of search engine crawling and indexing practices.