A robots.txt file tells search engine crawlers which URLs they can request from your site. It controls crawling traffic to protect your host from overload, yet it leaves existing pages in Google search results. If you want to remove a URL from search results or protect confidential data, you need other tools.

What to check before changing crawl settings
Before modifying your crawl rules, gather a few basic items to avoid locking out the wrong sections of your site:
- Access to your web server root directory or CMS file settings.
- An active property in Google Search Console.
- Access to your security CDN or hosting firewall settings.
- A clear list of URLs or paths that you want to restrict.
What a robots.txt file blocks
A crawl directive acts as a signpost for automated visitors. When Googlebot visits your domain, it first downloads the text file located at your root domain. If you add a Disallow rule for a directory, compliant crawlers skip requesting those URLs. This saves server bandwidth and keeps crawlers focused on your primary content.
A crawl block only stops crawlers from downloading file content. It does not stop Google from displaying the URL in search results. When external websites link to a disallowed URL, Google knows the address exists. Because Googlebot cannot fetch the page to read its content, the search result often appears with a bare URL and no snippet text.
How crawl rules differ from noindex and firewalls
Many website owners mix up crawler directives with indexing tags and network firewalls. Each tool operates at a different layer of your infrastructure.
A noindex directive asks search engines to drop a page from search results. To read a noindex meta tag or HTTP response header, Googlebot must fetch the page. If you block the page in your crawl file and add a noindex tag, Googlebot never downloads the HTML. Google never discovers the instruction, and the URL can stay in search results.
Web application firewalls and security filters operate before crawl rules matter. A firewall can reject requests based on IP reputation, request rates, or bot signatures. A page can load smoothly in your browser while Googlebot receives a 403 Forbidden or 429 Too Many Requests response. In that case, your text file is ignored because the crawler never reaches your server.
Specific crawler tokens also do not control general search indexing. Rules for Google-Extended control whether Google can use your content to train Gemini models and to ground answers in Gemini Apps and Vertex AI. As explained in our guide on AI search crawler access, blocking Google-Extended does not alter your standing in Google Search results.
Steps to inspect what Google can fetch
Follow these four steps to verify whether your pages are reachable and eligible for search results:
- Read your file directly. Open the file at your root domain in a browser (for example,
example.com/robots.txt). Check whether a broad rule likeDisallow: /appears underUser-agent: *orUser-agent: Googlebot. - Test individual URLs in Google Search Console. Use the URL Inspection tool, following the steps in our guide on how to check Google indexing. The live test displays whether Googlebot can reach the page or hits a directive block.
- Run an automated site audit. An on-page SEO audit fetches each page and reads the HTML that the server sends, as a crawler does on its first pass. It reports pages that are blocked from indexing, 4xx pages, and redirects. It does not run JavaScript.
- Check your server logs and firewall rules. If Search Console reports fetch errors while your crawl file has no blocking rules, inspect your firewall activity for blocked Googlebot IP ranges.
As an illustrative example, during migrations a staging directive can accidentally be deployed to production. In one example scenario, a team wondered why newly published articles failed to show up on Google. Their root file still contained a rule blocking all crawlers from the root directory. Because the site was closed to crawlers, Googlebot never fetched the content. Removing that single line allowed normal crawling to resume.
Common crawl configuration mistakes
Directives are simple text lines, but small oversights cause widespread indexing trouble. Watch out for these four errors:
- Hiding private information: Directives in this file are public. Anyone can view the URL to see which paths you want to hide. Use password authentication or server-level access controls for sensitive assets.
- Pairing disallow with noindex: If you want a page out of Google Search, allow crawling and include a
noindextag. Do not disallow the URL in the text file at the same time. - Blocking CSS and JavaScript: Googlebot needs stylesheet and script files to render pages properly. Blocking asset folders can prevent Google from confirming your layout is mobile-friendly.
- Expecting automatic indexing: Allowing crawler access gives permission to fetch a URL, but it does not force Google to index the page or rank it.
Frequently asked questions
Does blocking a page in the crawl file remove it from Google?
Blocking a URL in your crawl file prevents Googlebot from downloading its content, but it does not remove the URL from Google Search. If other websites link to that address, Google can still index the URL using link signals and display it without a snippet description. To remove a page completely, keep it crawlable and apply a noindex tag instead.
Why does Google index a disallowed page?
Google indexes disallowed pages when external or internal links point to them with clear anchor text. Because your directives prevent Googlebot from loading the page, the crawler cannot read your on-page text or meta tags. Google relies solely on external references to understand the URL, which often results in an empty search result snippet stating that no information is available.
Can a firewall block Googlebot even when the text file allows access?
Yes. A web application firewall or hosting security filter evaluates incoming server requests before your directives are even checked. If your security rules misidentify Googlebot IP addresses or rate-limit requests too aggressively, the server returns 403 or 429 response codes. The page will load fine for human visitors in a browser while remaining completely inaccessible to search crawlers.
To see how your pages perform after resolving crawl issues, you can monitor your keyword positions with RankMySEO.
Keep reading