Robots.txt Generator
Build a valid robots.txt file to control how search engines crawl your site, then copy or download it.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
A robots.txt file lives at the root of your domain and tells search engine crawlers which URLs they can request. Get it wrong and you can accidentally hide your whole site, or expose pages you meant to keep private, so it's worth generating cleanly.
Pick a preset, optionally block specific paths, add your sitemap, and download the result straight to your project's root directory.
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain-text file placed at the root of your domain (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not access. It's a key part of technical SEO.
Where do I put the robots.txt file?
It must live at the root of your domain, like https://yoursite.com/robots.txt. Crawlers look for it there and nowhere else. Download the file from this tool and upload it to your site's root directory.
Does robots.txt hide pages from Google?
Not exactly. Disallow stops compliant crawlers from fetching a page, but a blocked URL can still appear in results if it's linked elsewhere. To keep a page out of the index entirely, use a noindex meta tag or HTTP header instead.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps search engines discover all your important URLs. This generator includes it automatically when you provide a sitemap URL.
More free tools
Metadata Viewer
See hidden data in your photos and PDFs: GPS location, camera info, author details.
SecurityPassword Generator
Generate strong, random passwords with customisable rules. Created locally, never sent anywhere.
DesignContrast Checker
Test colour combinations against WCAG accessibility standards in real time.
Need this built into your product?
HG Studio builds custom tools, automations, and web apps. Let's talk.
Work with HG Studio