Robots.txt Explained: What to Check Before Launch
The robots.txt file is small, public and easy to forget. During development, teams often use it to discourage crawling. At launch, that temporary rule can become a production problem if nobody checks it.
What robots.txt controls
A robots.txt file provides crawl instructions for user agents. It can allow or disallow paths and can point to sitemap locations. It does not act as an access-control system. A URL blocked from crawling can still be discovered through links or other sources, and sensitive information should never rely on robots.txt for protection.
Use authentication and proper authorization for private content.
Where the file belongs
The file is served from the root of a host, such as https://example.com/robots.txt. Rules apply to that host and protocol context. If a project moves from a preview domain to a custom production domain, confirm the production host serves the intended file.
A missing file is not automatically a problem. A broken or overly restrictive file can be.
The launch-blocking pattern
One of the most damaging simple mistakes is a production file that contains a broad disallow rule such as Disallow: / for the relevant crawler. This is often intentional on staging. Before launch, compare production rules with the actual indexing plan.
Also check CMS and framework settings that may add a noindex robots meta tag independently of robots.txt.
Sitemap references
You can include one or more Sitemap: lines with absolute URLs. This makes the sitemap easier for crawlers to discover. The referenced sitemap should itself load successfully and contain canonical URLs from the current production domain.
Do not keep obsolete sitemap references after a migration.
Do not use robots.txt to hide secrets
Because the file is public, disallowed paths can reveal interesting locations to anyone who reads it. Private admin panels, API keys, internal files and personal data need real security controls. Robots directives are about crawler behavior, not confidentiality.
Treat any URL available without authorization as public, regardless of whether robots.txt asks bots not to crawl it.
Automated checks can confirm whether a public signal exists and looks technically reasonable. They cannot replace human review of content, accessibility, security, business logic or legal requirements.
Check a website