A website can work perfectly for visitors while some AI search systems receive only part of its content. Prompt tracking, which involves asking AI assistants selected questions and recording whether your brand appears in their answers, can show how your brand is represented. What it cannot establish is whether those systems can retrieve information from your website.
Large language models (LLMs) aren’t browsers. When AI assistants retrieve information from the web, they rely on search indexes and other retrieval tools. What those systems receive depends partly on how the website is built, protected and maintained.
Why is AI search more challenging for enterprise websites?
In my experience, enterprise websites rarely behave as one uniform system. They often combine different content management systems (CMS), custom developments, integrations and page templates.
Even within the same platform, the way content is delivered can vary. Some pages provide their information immediately, while others depend on the browser to load important details afterwards.
Knowing which platform a website uses doesn’t necessarily tell you how every page behaves.
There are also infrastructure and security considerations. Content delivery networks (CDNs), firewalls and bot protection systems help keep websites fast, reliable and secure. Their configurations also determine how automated requests are handled.
This is where ownership often falls into a grey area. Marketing may be responsible for content and visibility, while engineering and security teams manage the systems that determine how content is delivered and who can access it.
With generative engine optimisation (GEO) becoming part of digital strategies, checking whether AI search systems can actually retrieve that content can easily fall between established responsibilities.
To understand where problems might occur, I separate the process into three questions: Can the system access the page, receive the important content and extract it in a usable form?
Let’s look at each one and why these technical considerations deserve a place in your GEO strategy.
Can AI search crawlers access your website?
AI companies use different automated systems to access web content. Some crawlers collect information for model training, others support search, while other tools retrieve pages when a user requests them.
For example, OpenAI’s crawler documentation distinguishes between OAI-SearchBot, used for ChatGPT Search, GPTBot, which collects material potentially used for training, and ChatGPT-User, used for certain user-initiated requests. These agents have different purposes and access rules. OpenAI allows website owners to manage search crawling separately from training crawling, while robots.txt restrictions may not apply to certain user-initiated ChatGPT-User requests.
Google’s AI features within Search, including AI Overviews and AI Mode, rely on Google’s established crawling and indexing infrastructure.
For Google’s AI Overviews and AI Mode, being accessible to Googlebot is only part of the picture. To appear as supporting links in these features, pages must also be indexed and eligible to appear with a snippet in Google Search. Google states that no separate technical optimisation is required specifically for these AI features.
How Website Security Can Affect AI Crawler Access
Enterprise security measures are designed to protect websites from unwanted traffic, including malicious automated requests. They are essential, but their configuration can also affect legitimate crawlers that an organisation intends to allow.
A person opening a website in a browser may pass the necessary security checks without noticing them. An automated system making a similar request can be treated differently.
Some protections require browser capabilities, such as executing JavaScript or completing a verification challenge. A crawler unable to complete those checks may receive a blocked response instead of the intended page.
Cloudflare’s documentation on browser challenges illustrates how these restrictions work.
Cloudflare also provides AI Crawl Control, allowing website owners to monitor AI crawler activity and manage access. However, a crawler permitted by one policy may still be restricted by another firewall rule.
Another element worth reviewing is robots.txt, a file that tells compliant crawlers which parts of a website they should avoid. Unlike a firewall, it doesn’t physically prevent access, but an overly broad rule could unintentionally exclude crawlers used for AI search.
A technical SEO lead can bridge the gap between content, marketing and engineering, working with these teams to develop a GEO strategy that aligns business priorities with how AI systems access and retrieve information from the website.
This collaboration also helps ensure that crawler access reflects the organisation’s security requirements and its AI search objectives, distinguishing intentional restrictions from unintended barriers.
Does the AI system receive the content visitors see?
Getting past the security layers is only the first step. The next question is whether the system receives the same important information that a visitor sees in a browser.
When a server responds to a page request, it sends HTML, the code containing the page’s structure and, depending on how it’s built, its content.
Sometimes that content is prepared on the server before being delivered. Other times, the browser executes JavaScript to load or assemble parts of the page afterwards.
These approaches include server-side rendering (SSR) and client-side rendering (CSR). Other techniques, such as static generation and hybrid rendering, can also deliver important content in the initial HTML or load it later through JavaScript.
The illustration below simplifies how different rendering approaches can affect what AI retrieval systems receive.
Consider a public product page. Its description might already be present in the initial HTML, while technical specifications or compatibility information are loaded afterwards through JavaScript.
A visitor using a browser sees both. However, a system processing only the initial HTML would receive the description without the additional details.
Google Search can execute JavaScript when processing pages for indexing. That doesn’t establish that other retrieval mechanisms, including those supporting ChatGPT, process pages in exactly the same way.
For this reason, the important distinction is which information is included in the initial HTML and which depends on additional browser processing. This can vary between page types and even between individual elements on the same page.
Some retrieval tools also convert HTML into a simplified text version, removing elements such as scripts, navigation and styling. Comparing the original HTML, the content rendered in a browser and the text extracted by a retrieval tool can help identify where important information is lost or becomes inaccessible.
For essential public content, technical SEO and development teams can evaluate any necessary improvements while taking the website’s existing architecture and performance requirements into account.
How can digital teams turn these findings into action?
The starting point is to establish which public pages and information matter most to the business, and what those pages need to make accessible to AI search systems. This gives the teams involved a common objective when evaluating technical improvements.
A solid foundation for GEO starts with a website whose relevant public content AI systems can access, retrieve and understand.
Bringing technical SEO, marketing, engineering and security together is what makes this possible within an enterprise environment, aligning AI search objectives with the website’s existing architecture and technical requirements.
Looking for a GEO partner?
I help enterprise teams assess their website’s AI search readiness and turn technical findings into practical improvements. Get in touch to discuss how I can support your GEO initiatives.