How Llms.txt And Robots.txt Affect AI Crawlers

None of these are traffic numbers, which is the uncomfortable part. Much of the value in this channel arrives without a click and shows up weeks later as somebody who already knew what you did before they contacted you.

There is also a mechanical problem. Manufactured mentions tend to be uniform in language and timing, which is exactly the pattern that gets discounted. The effort produces a body of sources that agree suspiciously well and carry less weight than a smaller number of genuine ones.

Include the constraints too. The job size you turn down, the sector you do not serve, the situation where a competitor is genuinely the better answer. Those are the statements that get quoted, and an agency will not invent them for you.

Blocking these is therefore not one decision. Turning away a training crawler is a defensible editorial position. Turning away the agent that fetches pages at answer time removes you from answers entirely, and the two are frequently confused.

State What You Sell in Concrete Terms Price range, lead time, geography, capacity, what you decline. This feels commercially sensitive and it is the material that makes your pages quotable, so an agency that does not have it will write vague content by necessity.

A weak brief produces a generic proposal, and a generic proposal produces a generic engagement that spends the first two months discovering things you already knew. The brief is the cheapest lever you have over the quality of the work.

What Honest Reporting Contains The prompt set, versioned and unchanged since last month. The raw answers, kept in full rather than summarised. Which competitors were named. Which sources were cited. What work was done. What moved, and the specific claim about which work caused it.

Every search marketing agency now offers this service. Some of them have built genuine capability, and some have added a page to their site and a line to their proposal template. From the outside the two look identical, because the vocabulary is easy and the results are hard to verify.

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

Keep a record of every correction you request and its outcome, including refusals. It gives you a realistic picture of which sources are worth approaching again, it prevents the same request being sent twice by different people, and it turns an activity that usually feels like shouting into a void into something with a measurable acceptance rate.

The Broader Lesson Every stage of this has punished the same thing, which is dependence on a single channel whose terms you do not set. Featured snippets did it, each core update did it, and this is doing it again with more force.

The complication is that AI systems use several distinct agents for different purposes. One may crawl for training corpora, another may fetch pages live when composing an answer, and a search provider's traditional crawler may feed both search results and an AI summary.

One practical note on prioritising outreach. Sort your citation list by frequency and start at the top, not at the most prestigious name. A directory that appears in half your category's answers is worth more than a publication that impresses your board and has never been cited once. This is the point at which visibility work and conventional public relations objectives diverge, and it is worth saying out loud before the two budgets start competing.

Finally, pay attention to how they talk about their existing clients. Somebody who describes a client's category accurately, names the specific constraint that made the work difficult, and mentions something that did not work has actually done the job. Somebody who describes every engagement as a success in identical language has either been unusually lucky or is describing a template.

A reasonable formulation: after two quarters, we expect movement in mention rate on buying intent prompts, improvement in the accuracy of how we are described, and new citations from the sources our baseline showed matter. If none of those move, we will treat the approach as unsuccessful.

This matters because the prompt set is built from it, and a prompt set written from segment language measures your positioning rather than your market. If your brief says mid market operations leaders, the prompts will use that phrase and no buyer ever will.

The decision that almost never makes sense for a commercial business is blocking the agents that fetch pages when composing answers. That is the mechanism by which you get recommended, and turning it off is the equivalent of declining to be listed anywhere, taken quietly, usually by accident.

That is an unglamorous conclusion and it has held through every disruption in this space so far. Fix your foundations, spread your discovery routes, and treat any plan that requires a single channel's rules to stay fixed as a bet rather than a strategy. ai seo agency