Most websites still have one front door. A visitor lands, reads a page, fills a form, maybe buys something. Search engines crawl the same public pages, then hopefully send a bit of traffic back. That was the bargain. Messy, imperfect, but familiar. AI has made the bargain awkward. A bot visiting your site might be indexing your content for search. It might be collecting material to train a model. It might be an agent acting for a real person who wants to compare prices, book a call, check availability, fill a basket, or pull information into another system. Those are not the same thing. Treating them as one big lump called "bots" is no longer good enough. Cloudflare has just made this split much more visible. Its new AI traffic controls let site owners manage automated visitors by behaviour: Search, Agent and Training. Search is about indexing and returning references. Agent traffic is automation acting on behalf of a person in real time. Training is a crawler taking content to improve or fine tune a model. That distinction matters. If a search crawler indexes your service page and sends people back, that can be a fair trade. If an AI answer engine reads your work, gives the answer elsewhere and sends no visitor, the trade changes. If a training crawler absorbs your original content into a model, that is different again. If a customer's agent visits your site to complete a task, blocking it might cost you a sale. This is where most business websites are behind. They were built for humans, then tuned for Google, then patched with a chatbot. Very few were designed with a clear view of what automated systems are allowed to read, store, quote, reuse or do. That is going to become a normal part of the website brief. Not someday. Now. The obvious reaction is to block everything. I get the instinct. Nobody wants their work scraped, remixed and monetised somewhere else while they pay for the hosting and the expertise. But blocking everything is blunt. It can also be self-defeating. Small businesses still need to be found. Local firms, consultants, agencies, software companies, venues, trades, ecommerce brands, charities, publishers. They need their useful pages to show up when a customer asks an AI assistant for options. They may want search and answer systems to reference them. They may even want buyer agents to complete certain jobs on behalf of customers. What they do not want is unlimited extraction with no attribution, no control and no commercial return. So the question becomes more specific: what do we want AI systems to do with this site? That is a better question than "How do we rank in AI search?" Ranking is only one piece of it. The bigger issue is access. A practical AI access policy should answer a few basic questions. First, what should search systems be allowed to index? For many businesses, the answer is most public service, product, location and educational content. If you want to be discoverable, machines need clean, structured, current information. That means clear pages, good schema, source-backed claims, FAQs where they help, and proof that survives being quoted out of context. Second, what should agents be allowed to do? Reading is one thing. Acting is another. An agent asking whether you serve Manchester is low risk. An agent booking a paid appointment, changing an order, requesting a refund or submitting a quote form is not low risk. Those actions need authentication, rate limits, confirmations, logs and probably human approval at first. Third, what should training systems be allowed to use? This is where businesses need to decide what is genuinely public, what is commercially valuable, and what they would prefer not to have absorbed into someone else's model. There is no single correct answer. A restaurant menu is not the same as a paid research archive. A public FAQ is not the same as a proprietary operating manual. Fourth, what content use is acceptable? Cloudflare is testing content-use signals around immediate use, reference use and full reproduction. That is the right shape of conversation. A system briefly reading a page to answer a user's live question is different from a system storing it, summarising it, reproducing it and never sending anyone back. Fifth, how will you know what happened? This is the boring bit and therefore the useful bit. Logs. Bot classifications. Referral data. Tool-call records. Form submissions by automated agents. Failed attempts. Approved actions. Rejected actions. If nobody can inspect it, nobody can govern it. This is where the agency model has to grow up. For years, the website brief has been design, copy, CMS, SEO, analytics and conversion. All still useful. None of it disappears. But it is not enough. The new brief needs a machine-access layer: which pages should be machine-readable, which content can be referenced but not reproduced, which automations are blocked, which agents can act, which actions require login or approval, which data sources are authoritative, what gets logged, and what can be rolled back. That does not sound as sexy as a homepage redesign. Tough. It is more important. The businesses that get this right will not just have prettier websites. They will have websites that can safely participate in agent-led discovery and transactions. They will be easier for AI systems to understand, safer for agents to interact with, and clearer about where the commercial boundaries are. The businesses that ignore it will drift into one of two bad positions. Some will leave everything open and complain later when their content, offers and data are used in ways they never agreed to. Others will lock everything down and wonder why they are invisible when customers start using agents and answer engines as their first port of call. Neither is strategy. It is panic in opposite directions. For a normal business owner, the first move does not need to be complicated. Start with an audit. List the public content that helps buyers make a decision. Make it accurate, structured and worth citing. List the content you do not want used for training or reproduction. Set the available controls properly, including robots and content signals where they apply. Separate low-risk information access from high-risk actions. If agents are going to submit forms, book calls, check stock or request quotes, decide what they can do without approval and what needs a human in the loop. Then measure it. Because this is not a one-off settings job. It is an operating layer. The machines visiting your site will change. Their behaviours will change. The commercial model around content, search, agents and payments will keep moving. Your rules need to be reviewed like any other part of the business system. This is also why "we need better AI SEO" is too small a brief. Yes, you need to be visible. But visibility without control is just exposure. Control without visibility is just hiding. The work is to decide what should be discoverable, what should be usable, what should be protected, and what should be actionable. That is the next website conversation. Not just "can people find us?" Now it is: when machines find us, what are they allowed to do?
X Post (ready to use)
> Your website now needs an AI access policy. > > A search crawler indexing your page ≠ a training bot absorbing your content ≠ a customer agent booking a call. > > Cloudflare just made that split official. The next website brief is: what are machines allowed to do when they reach your site? > > Visibility without control is just exposure. > [Link]
LinkedIn Teaser
Most websites were designed for one type of automated visitor: Google.
That model is breaking down fast. AI has introduced at least three distinct categories of machine visitor , search crawlers, training bots, and agents acting on behalf of real customers , and most business websites have no policy for any of them. Cloudflare's new AI traffic controls make this concrete. Site owners can now manage automated visitors by behaviour: Search, Agent, and Training. That taxonomy is already reshaping how serious operators think about their web presence. The old web bargain was simple enough: let search engines crawl your site, and with luck they send visitors back. AI has made that deal weird. An AI answer engine that reads your page, gives the answer somewhere else, and sends no one back , that's a different trade. A training crawler that absorbs your content into a model without attribution or payment , different trade again. A customer agent that visits your site to complete a task on behalf of a real buyer , one you might want to encourage. The businesses that figure this out will have websites that are discoverable by machines, safe for agents to interact with, and clear about where the commercial boundaries sit. The ones that don't will oscillate between leaving everything open and blocking everything , panic in both directions. Read the full piece to see what a practical AI access policy looks like for a normal business. [Link]
last30days signal confirms timeliness: Cloudflare AI traffic controls (July 1 2026), robots.txt AI crawler management guides trending, and HTTP signature-based agent identification are all live topics. Strong signal. Strong draft. Ready to publish.
FAQ
What is an AI access policy for a website?
An AI access policy is a practical set of rules for what automated systems can read, reference, reuse, train on, or do when they interact with a website.
Should a business block all AI crawlers?
Usually no. Some automated visitors help buyers discover and evaluate a business. The better move is to separate useful search and agent activity from training, scraping and high-risk actions.
What should be included in an AI access policy?
Start with indexable content, training-use boundaries, agent actions, authentication rules, approval gates, content-use signals and logs that show what automated systems attempted or completed.
Why does this matter for SEO and AEO?
Answer engines and buyer agents need accurate, structured, attributable information. A good access policy keeps that discoverability while protecting commercially sensitive content and risky actions.