The technical AEO checklist: what to verify before anything else
Eight checks: AI crawlers allowed in robots.txt; your CDN or firewall not silently blocking them; content readable without JavaScript; fast responses; clean heading structure with self-contained sections; correct structured data; AI-hallucinated URLs redirected instead of 404ing; and analytics that can actually see AI referrals. Most businesses fail at least two of these without knowing it.
What are the technical requirements for AI search visibility?
Crawler access: the five-minute check that fails most often
Confirm robots.txt does not disallow GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended. Around 6% of sites block GPTBot, often via a boilerplate robots file someone copied years ago. Then test with real user agents, not just the file: Cloudflare now blocks AI crawlers by default, and a firewall rule can return errors to bots while the robots file looks permissive. Fetch your key pages as each bot and confirm an HTTP 200 with full content — we re-run this check on our own site after every infrastructure change.
JavaScript rendering: the silent killer
ChatGPT's crawler does not execute JavaScript. A single-page app that builds its content client-side serves AI crawlers a shell — a title, a script tag, and none of the words. This is the most common and most expensive AEO defect, because the site looks perfect in every browser while being effectively blank to the systems you want citing it. The fix is server-side rendering or build-time prerendering of every route that carries content. Judge it the only honest way: fetch the raw HTML with JavaScript disabled and count the words of real content.
Structure, speed, and the shape of a citable page
Slow pages get dropped from live retrieval before they are ever scored, so response time is a visibility factor, not a nicety. Structure matters because retrieval chunks pages: one clear H1, H2 sections that each make sense read alone, the answer at the top of each section rather than built up to. Keep structured data honest and current — Organization, Service, FAQ — less because schema wins citations (evidence is mixed) and more because it states facts about your business unambiguously.
One H1; H2 sections that stand alone when quoted out of context
The answer in the first two sentences of the page and of each section
Entities named: your firm, the product, the number — not "our solution"
A visible, real last-updated date you actually honour
The 404s nobody watches, and measurement
AI assistants hallucinate URLs — plausible paths on your domain that never existed — and send roughly 2.9 times more 404 traffic than Google does. Pull AI-referred 404s from your logs and 301 each to the closest real page; that is recovered demand at zero acquisition cost.
Finally, make the channel visible: a custom analytics channel grouping matching chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai referrers, plus an "how did you hear about us" field on intake. Referrer data undercounts — several platforms strip it — which is exactly why the intake question is the ground truth.
Should I block AI crawlers to protect my content?
If AI assistants are a channel you want customers from, blocking their crawlers is self-inflicted invisibility. The trade is content for presence. For a business site whose pages exist to win customers, allowing reputable AI crawlers is almost always the right call.
What about llms.txt?
It is a proposed convention for handing AI systems a clean text version of your site. No major provider commits to reading it today. Ours costs nothing to generate from the same data that builds our pages, so we ship it — but treat it as a lottery ticket, not a lever.
How do I check what an AI crawler sees on my site?
Fetch your page with curl using the bot's user agent and JavaScript off, and read what comes back. If the words your customers need are not in that response, they are invisible to ChatGPT's crawler. Repeat for your five most important pages; the homepage is usually the worst offender.
Does page speed really affect AI citations?
Yes — live retrieval works on a time budget, and pages that respond slowly get dropped from consideration before content quality is ever evaluated. The bar is not exotic: fast server response and no multi-second render dependency.
Last reviewed 23 August 2026