What robots.txt does
It's a small text file at `yoursite.com/robots.txt` that tells search-engine crawlers which parts of your site they're allowed to access. It's guidance for well-behaved bots, not a security measure — it doesn't hide pages, it just requests they aren't crawled.
Generate it
- Use the Robots.txt Generator to build the rules and copy the file.
- Upload it to your site's root so it's reachable at `/robots.txt`.
The basics of the syntax
- `User-agent: *` — applies to all crawlers.
- `Allow: /` — lets them crawl everything.
- `Disallow: /private/` — asks them not to crawl that folder.
- `Sitemap: https://yoursite.com/sitemap.xml` — points crawlers to your sitemap.
Pair it with a sitemap
robots.txt says what *not* to crawl; a sitemap lists what you *want* crawled. Build one with the Sitemap Generator and reference it from robots.txt.
A warning
Don't accidentally `Disallow: /` your whole site — that blocks all crawling and removes you from search.
Private
The file is generated in your browser.