Controls the behavior of search engine bots and may contain most of the directives usually found in robots.txt file.

Reference...

We have seen 6732 websites setting the X-Robots-Tag HTTP header and 103 unique values of this header.

Fully automated RESTful API is now available. Subscribe for your free trial today!