Where do I put the robots.txt file?

At your domain root so it loads at yourdomain.com/robots.txt. On WordPress you can add it via your host's file manager or an SEO plugin.

Does robots.txt keep a page out of Google?

Not reliably. Blocking crawling can still leave a URL indexed without a snippet. To keep a page out of results, use a noindex meta tag instead.

Is this robots.txt generator free?

Yes. It builds a valid file in your browser with no login. Copy or download it and upload to your site root.

robots.txt Generator

This free robots.txt generator builds a valid file that tells search engine crawlers which parts of your site to crawl. Choose to allow crawling with specific disallowed paths or block everything, add your sitemap URL, and copy or download a ready-to-upload robots.txt.

Key takeaways

robots.txt guides crawling; it is not a security or reliable de-indexing tool.
On WordPress, disallow /wp-admin/ and cart/checkout, but never CSS or JS.
Always include a Sitemap: line so engines discover your important URLs.

What does robots.txt do?

robots.txt is a plain-text file at your domain root that gives crawlers rules about which paths they may or may not request. It is a crawling guideline — not a security control — and it is the standard place to point crawlers at your sitemap.

What should a WordPress robots.txt disallow?

/wp-admin/ — the admin area.
Cart, checkout and account pages on WooCommerce stores.
Internal search results and thin, parameterized URLs.

Avoid blocking CSS, JS, or your uploads folder — crawlers need those to render pages correctly.

Always link your sitemap

Adding a Sitemap: line helps search engines discover every important URL. If you run nativeWP's native blog, your sitemap already updates automatically as posts publish. nativeWP is an AI content automation platform for WordPress that bulk-generates SEO articles with your own OpenRouter API key and auto-publishes or schedules them to WordPress and Shopify.

Frequently asked questions

Where do I put the robots.txt file?: At your domain root so it loads at yourdomain.com/robots.txt. On WordPress you can add it via your host's file manager or an SEO plugin.
Does robots.txt keep a page out of Google?: Not reliably. Blocking crawling can still leave a URL indexed without a snippet. To keep a page out of results, use a noindex meta tag instead.
Is this robots.txt generator free?: Yes. It builds a valid file in your browser with no login. Copy or download it and upload to your site root.

nativeWP is an AI content automation platform for WordPress that bulk-generates SEO articles with your own OpenRouter API key and auto-publishes or schedules them to WordPress and Shopify.