What the X-Robots-Tag header is
X-Robots-Tag is an HTTP response header that accepts the same directives as the meta robots tag, such as noindex, nofollow and nosnippet. Because it lives in the HTTP response rather than in HTML, it works for any file type.
You can also aim it at one crawler, for example X-Robots-Tag: googlebot: noindex.
When to use it
Use the header for files that cannot contain a meta tag: PDFs, Word documents, images and feeds. It is also handy for applying a rule to a whole folder or file type in server configuration, for example noindexing every PDF, or for keeping an entire staging host out of search engines.
Common mistakes
Staging configuration that sends X-Robots-Tag: noindex sometimes ships to production and quietly removes a site from Google; the header is invisible in the page source, so check the response headers after every deploy. As with any noindex, the URL must not be blocked in robots.txt or the crawler never sees the header.
Example
Noindex every PDF:
# nginx
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
# Apache
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>