How Llms.txt And Robots.txt Affect AI Crawlers

Check your robots file, then check your server logs for the relevant agents and see what status codes they receive. A site that returns a challenge to every non-browser request is invisible to this entire channel, and nobody involved will have thought of it as a marketing decision.

There is almost always a specific, findable reason for this, and it is rarely that the model dislikes you. Here are the causes worth checking, roughly in the order that they tend to be responsible. answer engine optimization

Month Two: Corrections and the First Rewrites The work should now be concentrated on the recurring sources from the baseline. Expect a list of listings claimed, details corrected and errors submitted, with names and dates attached.

Keep a record of what you predicted as well as what you measured. Writing down at the start of a quarter what you expect to move, and then reading it back at the end, is the cheapest way to find out whether your model of this channel is any good. Most teams never do it, which is why the same confident explanations survive for years without ever being tested.

Which Publications Actually Get Cited Do not assume, and do not use prestige as a proxy. Run ten prompts in your category and count the cited domains, because the pattern varies enormously by industry.

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

What Should Not Have Happened Yet A large volume of new content. Twenty published articles by month three usually means the baseline was not used to direct the work, and the pages were commissioned before anyone knew which questions mattered.

If your organic impressions held steady while clicks fell, you have probably met this already. An AI generated summary now sits above the results for a large share of informational queries, answers the question in place, and leaves the ten blue links below it with less to do.

One measurement caution matters when reporting this internally. Search Console does not separate impressions where a summary appeared from those where it did not, so you cannot isolate the effect cleanly. What you can do is compare affected query types against unaffected ones over the same period, which controls for seasonality and for site wide changes and gives a defensible estimate rather than a guess dressed as a figure.

Being the Source Instead of the Casualty The summary cites sources, and being one of them is now a legitimate objective. The requirements resemble what earns citations anywhere else: a page that answers directly, contains specifics worth attributing, and is reachable and readable by a crawler.

Which to Fix First Work in that order, because the sequence is roughly cheapest to most expensive and each step is wasted without the one before it. There is no value in earning press coverage if the crawler cannot reach the page it points at.

Coverage in a publication nobody cites is still worth having for reach and credibility. It is simply not visibility work, and counting it as such is how a public relations budget gets defended for years without moving anything measurable in this channel. answer engine optimization

The Blocks Nobody Chose Most blocking discovered during audits was never a decision. A disallow copied from a template. A staging rule that survived a migration. A security plugin with an aggressive default. A content delivery network setting labelled bot protection with a switch nobody has looked at since launch.

Expect the timeline to be uneven. Crawler access can change what an assistant sees within days, because retrieval happens at answer time. Identity consistency takes longer, since scattered mentions have to be re-crawled before they join up. Third party coverage is slowest of all and is the part you control least directly, which is exactly why it is worth starting on it before you need the result.

Fragmented identity produces a specific symptom worth recognising: an assistant knows facts about you but attributes them vaguely, or confuses you with a similarly named business. The fix is dull consistency work across every place your name appears.

What robots.txt Controls It is a request, honoured by mainstream crawlers, that certain user agents avoid certain paths. It has no enforcement behind it and it does not secure anything, but the major providers respect it.

The third question matters most. A good answer names a cause, attaches a number and admits an alternative explanation. A weak answer describes activity in the language of effort without connecting it to anything observable.

None of this is a restoration of what was there before. It is an adjustment to a results page that now answers a portion of the questions itself, and the sooner the planning reflects that, the less painful each further change becomes. answer engine optimization