Organic AI visibility · Technical SEO for AI

Technical SEO for AI: make sure AI engines can reach, read and cite your site

Technical SEO for AI makes sure the crawlers behind AI engines can reach your pages, read them in plain HTML and extract clear answers.

It’s the least visible part of AI search optimization, and the one everything else depends on. If a crawler is blocked, the best content in your category won’t be cited.

Get your AI visibility audit

Contact Jarek

Included in every AI Search Retainer

Foundation · Growth · Authority. From $4,000/month.

The technical check is also part of the AI Visibility Audit, from $3,000.

Why it matters

Many sites block AI crawlers without anyone deciding to. A default CDN or firewall setting, a security plugin or an old robots.txt rule is enough.

AI engines also don’t all behave the same way. OpenAI, for example, runs separate bots: OAI-SearchBot decides whether your pages can appear in ChatGPT search, while GPTBot collects data for model training. You can allow one and block the other. Most site owners don’t know which ones they’re blocking.

And some pages are simply hard to read. If your key content only appears after JavaScript runs, or sits inside tabs, sliders and images, an engine may never see it.

What we do

What we check and fix

1. Crawler access

robots.txt rules for AI search and AI training bots, per company (OpenAI, Anthropic, Perplexity, Google, Microsoft).
CDN, firewall and bot-management settings (for example Cloudflare), which often block bots before robots.txt is even read.
Server logs, where available, to see which AI bots actually visit and which pages they request.

2. Rendering

Whether your key content is in the HTML the server sends, not only after JavaScript runs.
Content hidden in tabs, accordions, images or PDFs that should be plain text.

3. Page structure

One clear H1, logical H2/H3 headings, short paragraphs.
Answers near the top of the page, not below the fold.
Tables and lists where the content is a comparison or a set of steps.

4. Indexing and site health

Indexing, canonicals, redirects, duplicate pages and internal links on the pages that matter.
Page speed and Core Web Vitals on key templates.

5. Hygiene we set up, without overselling it

Schema

Organization, Person, Service, FAQ and Product where relevant. It helps search engines understand your site. An Ahrefs test of 1,885 pages found no significant lift in AI citations from adding it, so we don’t sell it as a citation lever.

llms.txt

We can add one. An Ahrefs study of 137,210 domains found 97% of llms.txt files received no requests at all, and Google says its AI features need no special files. It’s cheap, not a strategy.

What you get

Technical AI access report

Which AI bots can reach your site today, which are blocked, and where the block sits (robots.txt, CDN, firewall, plugin).

A decision table

For each bot: what it’s for (search or training), what we recommend, and what you decide.

Fix list in priority order, with effort per item and who does it (your developer, your host or us).

Implementation of the fixes we can make directly, and a written brief for the ones your developer needs to make.

Monthly monitoring as part of the retainer, because CDN and plugin updates can quietly change your settings.

Process

How it works

Access check

robots.txt, CDN and firewall rules, rendering of key pages.

Decision

You choose which bots to allow, with our recommendation per bot.

Fix

We fix what we can; your developer gets a brief for the rest.

Verify

We test access again after changes.

Monitor

Monthly checks inside the retainer.

Who it’s for

Sites on a CDN or with a security plugin that have never checked AI bot access.
JavaScript-heavy sites (single-page apps, headless builds) where content may not be in the initial HTML.
Businesses that blocked “AI bots” as a blanket rule and want to decide deliberately.

Who it’s not for

Teams that want a one-off technical SEO audit of the whole site with no AI angle. We do that too, under SEO audit. SEO audit →
Publishers whose goal is to keep content out of AI entirely. We can help set that up, but that’s a different project from getting cited.

Results

Proof

Technical SEO has been part of our work for 16 years, across 100+ clients.

Fintech · international

ZEN

+524% organic traffic.

Top-10 keywords from 122 to 1,044.

Healthcare · US

BestMind

+324% organic traffic in one year.

+87% Google Business Profile clicks.

Working with DatGrowth also allowed us to work better in improving our E-A-T (Expertise, Authoritativeness, Trustworthiness) and YMYL (Your Money Your Life), which is vital in the medical/pharmaceutical industry.
Tomasz Mierkiewicz, Baza-lekow.com.pl

FAQ

Technical SEO for AI FAQ

What is technical SEO for AI?

Technical SEO for AI makes sure AI crawlers can reach your pages, read the content in plain HTML, and extract clear answers from it. It covers robots.txt, CDN and firewall rules, rendering, page structure and indexing.

Should we block AI bots?

It depends on what you want. Some bots collect data for model training; others fetch pages for live AI search answers. Blocking the search bots can remove you from AI answers. If you want to be recommended by AI, don’t block them. You can still block training bots if you prefer.

Which AI crawlers should we allow?

For visibility in AI search, the search and user-triggered bots of the engines your buyers use, such as OpenAI’s OAI-SearchBot for ChatGPT search. Google AI Overviews and AI Mode rely on Googlebot. We give you a table per bot with a recommendation and let you decide.

Do we need llms.txt?

Not as a priority. Current evidence shows llms.txt files are rarely requested by AI crawlers, and Google says its AI features need no special files. We’ll add one if you want it, because it costs little. We won’t pretend it gets you cited.

Does schema markup help us get cited by AI?

Not on the evidence so far. An Ahrefs test of 1,885 pages that added structured data found no significant lift in AI citations. Schema still helps search engines understand your site, so we set it up as hygiene.

How do we know if AI bots can reach our site?

We test it: robots.txt, CDN and firewall rules, and how your key pages render without JavaScript. Where you can share server logs, we also see which AI bots actually visit. The report shows it bot by bot.

How long does this take?

The access check takes a few business days. Fixing a robots.txt rule takes minutes; CDN and rendering fixes depend on your developer and your stack.