Robots.txt generator

Generate robots.txt file for your website. You can copy the generated robots.txt file and paste it in the root directory of your website.

Rules (one directive per line)

Example: Allow: / or Disallow: /private

Sitemap URL

robots.txt

User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

About robots.txt

Robots.txt is a text file at the root of your website, for example https://yoursite.com/robots.txt. It is used to tell the search engines which pages to crawl and which to ignore.

Allow and Disallow lines

Allow means the path is open for crawling. Disallow means please do not crawl that path. One line per rule is the usual style. If you leave the rules box empty we put a simple Allow all so your site stays open by default.

Sitemap line in robots.txt

Adding your sitemap URL here is a common practice. Crawlers can find your sitemap from this file. Use our sitemap generator on this site if you still need to create the XML file.

FAQ

Will Disallow remove my page from Google?: Disallow reduces crawling of that path. For removing a page from search results people often use noindex on the page itself or Search Console removal tools. robots.txt alone does not hide content from everyone.
Where do I upload robots.txt?: Upload to the public root of your domain so the URL is exactly https://yourdomain.com/robots.txt. File name must be robots.txt in small letters.
Can I block the whole site by mistake?: Yes if you write Disallow: / on the wrong line. Always read the preview output here before you save on server. Keep a backup of your old file.