What robots.txt does
Robots.txt is a plain-text file served at /robots.txt on every host, for example https://example.com/robots.txt. Before a well-behaved crawler such as Googlebot fetches a page, it reads this file and checks whether the URL is allowed for its user agent. The rules follow the Robots Exclusion Protocol, standardised as RFC 9309 in 2022.
Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google, usually with no description, if other pages link to it. To keep a page out of search results, let it be crawled and add a noindex directive instead.
How the rules are matched
A file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the group that names it most specifically and falls back to the User-agent: * group only when no group names it.
Inside a group the longest matching path wins, and Allow wins a tie. Paths are case-sensitive, * matches any run of characters and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks every PDF. Each host and protocol needs its own file: blog.example.com does not inherit the rules of example.com.
Mistakes that cost traffic
The classic one is Disallow: / copied from a staging site, which blocks the whole website. Others are blocking CSS and JavaScript files that Google needs to render pages, blocking a page you also marked noindex (so the noindex is never seen), and expecting Crawl-delay to slow Googlebot, which ignores it.
Google reads only the first 500 KiB of the file, and a robots.txt that returns a 5xx error can make Google pause crawling of the whole site. List your XML sitemap with a Sitemap: line so crawlers find it without Search Console.
Example
A typical robots.txt:
User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml