Quick answer: Separate crawler access, search indexing, search previews and AI content guidance before changing your website controls. Robots.txt, noindex, Google-Extended and llms.txt address different purposes. Choose the outcome you want, consult the relevant platform documentation, test a limited change and monitor it. None of these files should be used as protection for private customer information.
Publishers now face questions that sound similar but need different answers. One business wants useful guides to appear in search while limiting particular model uses. Another wants to remove a staging site from public discovery. A third wants documentation to be easier for agents to navigate. Copying a single “block AI” snippet across all three situations can produce the wrong result.
Platform documentation and the llms.txt proposal checked on 21 October 2026.
Start with the outcome rather than the file
Write the desired result as a plain sentence. For example: keep product documentation discoverable in Google Search, prevent public access to customer records, or provide an optional concise directory of documentation. These are different tasks with different owners. A developer cannot validate a rule sensibly if the business has not decided which result matters.
Record the affected URLs and products, including any subdomains. A rule intended for a documentation area should not accidentally cover the whole store. Keep an approved copy of the current configuration before editing. Your technical SEO checklist should include this inventory so crawler decisions are reviewed alongside ordinary site access.
Understand what robots.txt is designed to do
Robots.txt communicates crawling preferences to compliant crawlers. It is publicly accessible, and its rules are not a login system. Google’s robots.txt introduction explains the scope and limitations. A crawler preference should not be confused with a guarantee that an address can never be discovered elsewhere.
Keep customer records, private enquiries and administrative exports behind appropriate authorization. Publishing their paths in a robots file does not make the contents private. For a staging environment, use a deliberate access restriction and verify it from a signed-out browser. Security decisions should not depend on every visitor identifying itself honestly as a crawler.
Distinguish Googlebot from Google-Extended
Google documents Googlebot as affecting Search and related search features. Google-Extended is a separate product token used to control specified Gemini training and grounding uses. It has no separate HTTP request user-agent string, and Google says it does not affect inclusion or ranking in Google Search. The exact current scope appears in its common crawler documentation.
This distinction matters when a team wants to keep a public page in ordinary search. Blocking the wrong crawler can undermine that goal. Review each platform’s documented controls independently instead of assuming that a token with “AI” in its description controls every AI-related experience the company operates.
| Control | Primary question | Important limitation |
|---|---|---|
| Robots.txt | May a compliant crawler fetch this path? | Does not protect private information |
| Noindex | Should a supporting search engine index this page? | The directive must be accessible to be read |
| Preview controls | What page information may appear in supported previews? | Check the relevant search platform’s rules |
| Google-Extended | May content support the specified Google model uses? | Does not control Google Search ranking |
| Llms.txt | Can agents find a concise guide to public content? | A proposal, not an access-control mechanism |
Use indexing and preview controls deliberately
When the goal is removal from indexing, review the search engine’s documented method rather than treating a crawl restriction as equivalent. Google explains noindex implementation separately. Blocking a page from crawling can stop the crawler from reading a new directive on that page. Check the actual response and the rendered or header-level instruction involved.
Preview decisions also need a defined goal. A publisher might want a particular page section excluded from a supported snippet while preserving the rest of the article. Google’s AI features documentation describes relevant Search preview controls. Review their tradeoffs before applying them widely, because reducing usable preview information can also change how a page is presented.
What llms.txt currently proposes
The llms.txt proposal describes a Markdown file that helps agents navigate useful content. Its current v2 page was modified on 10 August 2026. It discusses concise guidance, links to detailed material and Markdown alternatives for pages. Treat those recommendations as a documentation format, distinct from crawler permissions and search eligibility.
A useful implementation starts with an existing documentation need. If a software company has many public reference pages, a compact map may help clients find the right material. If a small business has only a few public pages, maintaining an additional file may offer less practical benefit than fixing unclear navigation and incomplete answers.
Keep optional machine-readable guidance factual and small
Include a short description of the organization or product, a limited set of authoritative resources and clear descriptions of what those resources contain. Point to maintained pages. Do not fill the file with repeated keywords, fictional credentials or claims that agents must recommend the business. It should help a reader find information, not attempt to dictate an answer.
If you publish Markdown alternatives, maintain the same substantive facts as the public HTML version. A second format becomes a liability when it contains older pricing, obsolete setup instructions or different product limitations. Assign an owner and update it through the same release process as the original documentation.
Test a crawler change on representative URLs
Choose one public article, one product page, one restricted area and any relevant subdomain. Record their expected behaviour before making a change. Confirm that the robots file loads from the intended host, that the correct rules apply, and that important page resources remain available when the search platform needs them.
Use the platform’s inspection tools where available, and check real server responses. Inspect both the successful path and the path you intend to restrict. A test that confirms only that the homepage loads does not establish that a deeper product page remains accessible. Keep screenshots or response notes internally when they help explain a production decision.
Interpret logs without assuming a user-agent proves identity
Log entries can reveal which paths are requested, how often failures occur and whether a rule change coincides with a different crawl pattern. Treat a user-agent label as a claim until it is validated through the provider’s documented method. A visitor can send a familiar-looking user-agent string without being the organization it names.
For Google, use its documented verification guidance rather than manually trusting any request containing “Googlebot.” Keep operational checks proportionate to the problem. If the issue is excessive load, investigate request patterns and infrastructure as well as crawler settings. A broad block may hide the symptom while creating a separate discovery problem.
Plan a rollback and a maintenance owner
Check which layer serves the final robots response. A physical file, a WordPress plugin and a CDN rule may each appear to offer control, but the public response is what a visitor receives. Inspect that response after clearing the relevant cache. Record the host being tested, because a rule on the main domain does not establish what a separate subdomain serves.
Review response headers when a hosting or security layer changes. A restriction added for staging can remain after a site is copied into production. Conversely, a template-based directive may disappear when a new theme takes over. Include a public article, a media resource and a restricted page in the release review so the team can catch both unintended exclusion and unintended exposure.
Keep the operational discussion separate from speculation about a ranking change. If the observed problem is a blocked public path, fix and verify that path first. It is a concrete configuration issue. Evaluate later discovery and visibility using the search platform’s actual reports rather than assuming that a restored response produces an immediate ranking outcome.
Every significant crawler change should have a known previous configuration and a person who can restore it. Record why the change was made and what observations would justify rollback. For example, an unintended restriction on a public catalogue is a configuration problem, regardless of whether the original intention was sensible.
Review the policy when platform documentation changes or the website launches a new public section. Your technical SEO audit can include a periodic comparison between intended access and actual responses. Avoid adding several plugins that each rewrite the same robots output without knowing which layer ultimately controls the result.
Keep public answers useful whatever formats you support
Clear page titles, descriptive headings, readable text, accurate links and maintained source references remain practical improvements. An optional guidance file cannot repair a broken page or an answer that omits essential conditions. Review the on-page SEO checklist before assuming an additional machine-readable format is the missing element.
Define one source of truth for important facts and let other formats point to it or derive from it. This reduces inconsistencies for readers, search systems and agents alike. The best control setup is the one that achieves the intended access and publishing outcome and can still be understood by the team maintaining it.
Common questions about AI crawler controls
Will blocking Google-Extended remove my site from Google Search?
Google states that this token does not control Search inclusion or ranking. Check the separate Googlebot rules and the current documentation before changing either.
Does llms.txt stop a crawler from reading my pages?
No. It provides optional guidance about content. Use the relevant crawler controls and proper authorization for access requirements.
Can robots.txt protect customer data?
No. Restrict private information with authentication and authorization. A publicly accessible file listing preferences is not a security boundary.
Should every website publish Markdown copies immediately?
Assess the reader or documentation need and maintenance cost first. Avoid creating a second version that cannot stay accurate alongside the original page.
Join the discussion