How AI Search Crawlers See Your Site - and How to Optimize Accordingly

Most teams audit their site by looking at it. The page loads, the reviews render, the FAQ accordion opens, and everything seems fine.

The crawlers behind ChatGPT, Claude and Gemini work differently. A good portion of what you see never reaches them, and what never reaches them can’t be retrieved, cited or recommended.

Most AI visibility problems stem from that disconnect – when the content doesn’t make the trip from your server to the AI model. In this post, we'll cover how AI crawlers read a page and give you a two-minute test that shows you roughly what they get and what they don’t (which highlights what you need to fix first).

TL;DR

• AI crawlers generally don't render JavaScript the way Googlebot does. Anything loading client-side, including reviews, ratings and some schema, may never be seen.
  
  
• Turn JavaScript off in your browser. What's still on the screen is roughly what the bots get.
  
  
• Keep pages lean and front-load the important information. Google reads somewhere around 2 MB per page, and AI crawlers appear to favor what sits near the top.
  
  
• Think about cost to crawl vs. value on the page. Clear structure, descriptive headings, sensible internal links and fewer scripts all lower the cost.
  
  
• If you appear in 20% of answers and competitors appear in 80%, the fix usually isn't publishing more. Find out which pages get cited and why yours don't.
  

Crawlability Comes First

If a site can't be crawled, content quality matters much less. That's not a new idea, but AI search has raised the stakes on it. The crawlers doing the reading are less forgiving than the SEO crawler teams built their habits around.

AI bots don't render JavaScript like Googlebot

Googlebot will queue a page for rendering and come back for the client-side content. The crawlers behind ChatGPT, Claude and, in many cases, Gemini often won't. They take what's in the initial HTML response and move on.

That tends to hit the same things on every site, including review widgets and star ratings, pricing tables that populate from an API, tabbed or accordion content, and schema injected by a tag manager rather than served in the source. All of that is visible to you but invisible to the model.

The two-minute test

You don't need a crawler license to check this. Turn JavaScript off, either in your browser settings or under the settings panel in Chrome DevTools, then reload a key page. What's still on the screen is roughly what the bots see.

First, run it on your highest-intent pages like your main service or product pages, your comparison pages, and your pricing page. If the thing you most want quoted back to a buyer disappears when scripts are off, you’ve got something to fix ASAP.

Keep pages lean

Google reads about 2 MB per page. Some AI bots appear to weigh the top of the page more heavily, which means the choice to bury an answer below a hero video, a carousel and three testimonial blocks will hurt your AI visibility.

Lead with what matters. State what the page is about, who it's for and what the answer is in the first screen of content, then expand below.

Think in Terms of Cost to Crawl

A useful way to frame all of this is as a trade. On one side sits the cost, meaning how much effort it takes a machine to read the page. On the other sits the value, meaning how useful the page content is.

A page heavy with scripts, thin on headings and vague about its subject is expensive to read and light on payoff. A page that states its topic clearly, breaks it into labelled sections, and loads as text is cheap to read and worth the effort.

The things that lower the cost are straightforward: clear structure, headings that describe what follows, internal links that help a reader get somewhere useful, reasonable page size, less reliance on scripts to deliver the substance.

This is essentially solid SEO advice, now with a second, less patient audience.

Fix the Main Site Before Anything Else

There's a temptation, once a team starts caring about AI visibility, to go straight to off-site work and chase mentions, pitch publications, seed communities. That’s important work, and we've written about how to turn clarity into credibility through third-party citations, but it pays off much faster when the destination is friendly for AI search crawlers.

To make your site easier for AI search to read, tackle the following:

• Make the site easy to reach and use, with nothing in robots.txt or your CDN rules quietly blocking AI user agents

• Use lists, tables and FAQs where they genuinely help a reader, because they also happen to be the formats models lift most cleanly

• Write headings that say what's below them

• Ask one question of every page. Is this clear for a person and easy for a machine to read?

A page that passes the last question rarely has a retrieval problem. A page that fails it usually has several.

Find Where You're Missing

Technical fixes are easier to prioritize when you know which answers you're losing. The loop we run is short and repeatable.

Start by picking the key questions, drawn from what buyers ask before they contact you rather than from a keyword tool. Then see where you appear across the platforms popular with your audience. Check who gets cited in the answers where you don't. Compare that set against your competitors. Make a plan from what you find, and measure it on a schedule.

You’ll find useful insights in the comparison step. Say you show up in 20% of answers and a competitor shows up in 80%. Don’t just react by publishing more. Look instead at which pages are getting cited and why yours aren't.

Often the difference is mechanical, e.g. their comparison page loads as plain text and yours depends on a script.

We'd also flag the discipline point here. One AI answer is a snapshot, not evidence. Record the prompt, the platform, the date and the cited sources for every check, or you'll have no way to tell improvement from noise. There's more on that in our piece on folding AI visibility into the SEO audit you already run.

Start With What You Can Check This Week

Open your five most important pages with JavaScript off and note what disappears. Confirm the main body text and the internal links are present in the source. Clear the broken links and the missing titles. Check that nothing in robots.txt or your CDN configuration is blocking AI user agents.

That's a week of work that decides whether everything you publish afterward is reachable at all. Our AI and LLM search team runs this as the opening move on every engagement, because the content strategy that follows is only as good as what the crawlers can get to.

If you'd like a second set of eyes on how AI search sees your site, drop us a line.

FAQs

Do AI crawlers render JavaScript?

Generally not the way Googlebot does. The crawlers behind ChatGPT, Claude and Gemini often take the initial HTML response and move on, which means content loading client-side can be invisible to them. Reviews, ratings, tabbed content and tag-manager-injected schema are the usual casualties. Serving that content in the HTML source is the reliable fix.

How can I see what AI crawlers see on my site?

Turn JavaScript off in your browser settings or in Chrome DevTools, then reload the page. What remains on screen is roughly what the bots receive. Run the test on your highest-intent pages first, meaning your service, product, comparison and pricing pages.

If competitors appear in more AI answers than we do, should we publish more content?

Usually not as the first move. Look at which pages are getting cited in the answers you're losing and compare them with your equivalents. The difference is frequently structural rather than editorial, with their page loading as plain readable text and yours depending on scripts, or their answer being marked up where yours sits behind a click.

Interested in a free strategy call? Schedule time with us.