FREE TOOL
Robots.txt Generator
Compose a clean robots.txt from user-agent rules and a sitemap URL.
Use "*" for all crawlers.
User-agent: * Disallow:
How it works
Add user-agent groups and their allow/disallow paths, point at your XML sitemap, and the generator composes a clean, valid robots.txt as you type.
Copy the output or download it as a file ready to serve at your site root.
Everything runs in your browser. Nothing is sent to a server.
Frequently asked questions
What does Disallow with an empty value mean?
It means allow everything for that user-agent — the standard way to write a fully permissive group.
Should I list my sitemap here?
Yes. The Sitemap: directive is the simplest way to hand crawlers your sitemap without registering it anywhere.
Rules written. Now earn the crawl.
Crawlers follow links. BuildSEO AI builds verified backlinks from qualified publishers so they follow them to you.
Try BuildSEO AI