Fix guide

How to create a robots.txt file safely

About 3 min read Time to do it: 10 min Updated
Quick answerRobots.txt tells crawlers which paths to avoid. Start from a conservative rule set, test important URLs and never use it as access control.
On this page
  1. Is this your problem?
  2. Why this matters for SEO
  3. Common causes
  4. Step-by-step fix
  5. How to confirm it worked
  6. Mistakes to avoid
  7. Tools to use
  8. FAQ
  9. Related guides

Why this matters for SEO

Robots.txt tells crawlers which paths to avoid. Start from a conservative rule set, test important URLs and never use it as access control.

Search visibility depends on several systems agreeing: the page must be reachable, discoverable, technically eligible to be indexed, useful for the query, and supported by a clear internal site structure. Fixing one symptom while another system is blocking the page can produce little or no improvement.

Use this guide as a focused workflow rather than a checklist to complete blindly. Confirm the actual page and response first, make the smallest defensible change, then re-test the same evidence after deployment.

Is this your problem?

  • You need to keep crawlers out of admin or internal search pages.
  • You are unsure what your current robots.txt does.

Common causes

  • A leftover Disallow: / blocks the whole site.
  • CSS or JavaScript files needed for rendering are blocked.
Diagnosis order

Check access and HTTP status first, then robots and indexability directives, canonicalization, content/template quality, internal links and performance. This order prevents a low-priority content edit from hiding a technical blocker.

Step-by-step fix

  1. Choose a preset

    Open the Robots.txt Generator and pick the user-agent and a conservative preset.

  2. Add only needed rules

    Disallow low-value paths such as admin or cart pages. Do not block CSS, JavaScript or pages you want indexed.

  3. Add your sitemap

    Include your sitemap URL so crawlers can find it.

  4. Publish at the root

    Save as robots.txt and upload it so it loads at https://yourdomain/robots.txt.

  5. Test key paths

    Use the Robots.txt Rule Tester with your homepage, key pages and assets, as Googlebot.

How to confirm it worked

Important pages and assets are allowed, and private areas are protected by real authentication, not just by robots.txt.

Verification should use the same URL and comparable conditions as the original test. Check the HTTP response, rendered/source markup and the relevant Google Search Console report where applicable. A successful technical change does not guarantee immediate indexing; crawling and indexing can take time and Google may still choose not to index a page it considers redundant or low quality.

Mistakes to avoid

  • Using robots.txt to hide sensitive pages; blocked URLs can still be known from links.
  • Expecting Disallow to remove a page from search.

Tools to use

Robots.txt Generator
Robots.txt Rule Tester

FAQ

Does robots.txt stop a page from being indexed?

Not reliably. It controls crawling. To keep a page out of search, allow crawling and use a noindex directive, or protect it with authentication.

Where must robots.txt live?

At the root of the host, for example https://example.com/robots.txt. Each subdomain needs its own.

Check your own site now. The free SeOMeter tools run these checks on your own URLs, no sign-up needed.

Open the SEO tools

Was something unclear or out of date? Tell us and we will improve this guide. SEO checks are diagnostic guidance, not a promise of rankings, traffic or indexing.