You changed robots.txt on the server, but the public address still shows the old rules. Before waiting for Google to refresh anything, establish which version your own delivery stack is serving. The origin, a reverse proxy, a CDN and Google's cached copy are separate places where an older response may remain.

The first useful question is simple: does an ordinary public GET request receive the intended file on the exact hostname you are investigating?

Save the public response before changing anything else

Request the actual root-level robots.txt address and save both headers and body. This example uses a documentation hostname; replace it with the real hostname you administer.

curl -sS -D public-robots-headers.txt -o public-robots.txt 'https://www.example.com/robots.txt'

Read the status line and any Location header as well as the rules. A redirect, login page or HTML challenge is not the same result as a successful response containing the intended text file. If the address redirects, record the chain and inspect the final response.

Keep the hostname exact. Rules served from the apex domain are not automatically the rules for its www host, another subdomain or a different protocol. A correct file on the wrong host does not complete this check.

Compare the origin through an authorised route

If you operate the origin, compare its response using the approved internal endpoint or diagnostic path for that deployment. Preserve the intended Host header and, for HTTPS, the correct server name and certificate verification. Opening a bare IP address can reach a default virtual host and produce a misleading difference.

Do not disable origin access controls to run this comparison. Where direct access is restricted, the hosting provider's tools or an internal request may be the right way to inspect what the origin serves.

  • Origin and public both old: investigate the deployed file, application route or origin-side cache. Purging the CDN alone would merely fetch the old version again.
  • Origin new, public old: investigate the proxy/CDN cache and whether robots.txt is generated or rewritten at the edge.
  • Public new, Google's recorded copy old: the deployment may be complete; now check Google's fetch history rather than repeatedly editing the file.
  • Responses differ across hosts: verify each host's routing and rule source before assuming a cache delay.

Use cache headers as clues, then verify the body

Look for Cache-Control, Age, ETag, Last-Modified and any provider-specific cache-status header. Their presence and interpretation depend on the stack. A high Age value can support a cache explanation, but no single header proves that every cache layer is fresh or stale.

A URL with a random query string may use a different cache key. It can help isolate a problem, but seeing new rules at /robots.txt?check=1 does not prove the normal /robots.txt request has been corrected. Repeat the exact public URL after the change.

If a stale CDN object is confirmed, use the provider's targeted purge for the affected host and path, including any relevant redirect response. Confirm that the purge completed, then perform another GET and compare the file itself. Keep the response time and the changed rule in your notes. An accepted purge job is not yet evidence of a fresh public response.

Google's cache is a separate check

Google's robots.txt specification says it generally caches the file for up to 24 hours, may keep it longer when refreshing is not possible, and may adjust the lifetime based on Cache-Control. This is not a deadline by which affected pages must appear or disappear from search.

Once the public file is correct, inspect the last fetched version and time in Search Console's robots.txt report where available. Guangsuan's guide to robots.txt refresh timing versus search-index updates explains why those two outcomes need to be tracked separately. Allowing a page to be crawled does not guarantee indexing; blocking crawling is not a reliable way to remove an already known URL.

A robots.txt error is also different from an old but valid file. Google's treatment depends on the HTTP status, so avoid replacing the file with a 403 response as an attempted quick block. Keep the intended rules available and investigate the actual response path.

Close the incident with a version trail

A useful handover contains the exact host, the intended rule change, origin and public response evidence, any cache action and its completion time, and Google's last fetched version if you can inspect it. Mark missing access as “not checked,” rather than inferring a result.

That record gives the next operator a clear stopping point: delivery corrected, crawler refresh pending, or delivery still inconsistent. It also prevents an unnecessary whole-site cache purge or another configuration edit when the remaining delay is outside the delivery stack.