Guide · 9 min read
JavaScript and AI crawlers: what assistants can read on your site
The link between JavaScript and AI crawlers decides whether assistants can read your pages at all. Googlebot runs JavaScript before indexing. Several AI crawlers, according to the best public test available, read only the HTML your server sends. If your content appears only after scripts run, some assistants may never see it. This guide explains what is documented, how to check your own site, and how to fix it.

Two versions of every page
When a crawler requests a URL, your server returns an HTML document. That is the first version of the page. If the page relies on JavaScript, the browser then downloads scripts, calls APIs and builds more content. The result is the rendered DOM, the second version.
On a classic server-rendered site, both versions are almost identical. On a single-page application built with client-side rendering, the first version can be little more than an empty container and a script tag. Everything a person reads, from headings to prices, exists only in the second version.
A crawler that does not execute JavaScript only ever sees the first version. That is the whole problem in one sentence.
How Googlebot handles JavaScript
Google documents its process in three phases: crawling, rendering and indexing. Pages that return a 200 status go into a rendering queue, where a headless, evergreen version of Chromium executes the JavaScript. Google then indexes the rendered HTML. The page may wait in that queue for a few seconds or longer.
Google still recommends server-side rendering or pre-rendering, partly because it is faster and partly because, in its own words, not all bots can run JavaScript. Source: Google's JavaScript SEO basics.
This matters for AI search because Google AI Overviews and AI Mode draw on the Search index that Googlebot builds. If Googlebot can render your page, those features work from the rendered version.
JavaScript and AI crawlers: what is documented and what is tested
Here is the uncomfortable part. At the time of writing, OpenAI, Anthropic and Perplexity describe their crawlers and what each one is for, but their crawler pages say nothing about whether those bots execute JavaScript. We checked the official pages for each. So any claim about rendering comes from outside testing, not from the vendors.
The most cited test is a study published by Vercel and MERJ in December 2024, based on crawler traffic across Vercel's network. Its findings, summarized:
- GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI): did not render JavaScript in the test.
- ClaudeBot (Anthropic): did not render JavaScript. Both OpenAI and Anthropic crawlers did download JavaScript files, but as text, without executing them.
- PerplexityBot (Perplexity): did not render JavaScript.
- Meta-ExternalAgent, Bytespider and CCBot: did not render JavaScript either.
- Gemini: rendered fully, because it relies on Googlebot's infrastructure.
- Applebot: rendered JavaScript through a browser-based crawler.
Source: Vercel, The rise of the AI crawler. Treat this as a snapshot. It is one study, nearly two years old, and vendors can change their crawlers without announcing it. Newer crawlers such as Claude-SearchBot were not part of it, so their behaviour is not publicly confirmed.
Microsoft Copilot was excluded from the study because it has no separate user agent. Copilot grounds answers in Bing, and Bing announced in 2019 that Bingbot renders pages with an evergreen Microsoft Edge engine. That suggests Copilot benefits from rendering, but we have not seen a recent test that confirms it end to end.
One more distinction. Agents that drive a real browser on a user's behalf, such as the hosted browser Anthropic identifies as Claude-Agent, load pages like a person would, scripts included. They visit on demand. They are not the crawlers that build the indexes most answers come from.
Why client-side rendered content can be invisible
If a crawler reads only the server response, anything injected later is missing from what it stores. On real sites, the content that tends to disappear is often the content that sells:
- Product names, prices, stock and specifications loaded from an API after the page opens.
- Reviews and ratings pulled in by a third-party widget.
- FAQ answers, tabs and accordions whose text is fetched only on click.
- Navigation and internal links built with click handlers instead of real anchor tags with an href.
- Titles, meta descriptions and canonical tags changed by a script after load.
- Structured data added by a tag manager rather than written into the HTML.
The effect is uneven. Your page may look complete in Google and in Gemini, while ChatGPT or Perplexity describe your company from third-party sources, or not at all. Because the page works for people, nobody notices. This gap between JavaScript and AI crawlers is one of the first things we look for when a brand is strong in Google and weak in assistants.
How to test what AI crawlers see
You do not need special software. These checks are quick. Test one page of each type: home, category, product or service, article.
- Compare view-source with the rendered DOM. In Chrome, open view-source:yourdomain.com/page and search for a sentence from the main content. Then open DevTools and search the Elements panel. If the sentence is only in Elements, it depends on JavaScript.
- Turn JavaScript off. In DevTools, open the command menu, type 'Disable JavaScript' and reload. What remains is roughly what a non-rendering crawler gets.
- Fetch the page with a bot user agent. Run curl -A "GPTBot" https://yourdomain.com/page and look at the HTML that comes back. Repeat with OAI-SearchBot, ClaudeBot and PerplexityBot. This also shows whether your CDN serves crawlers a challenge page or a 403. Keep in mind that a spoofed user agent does not come from the vendor's IP ranges, so some firewalls treat it differently from the real bot.
- Read your server or CDN logs. Filter by the crawler names and check status codes, which URLs they request and how often. If they fetch many script files but few content URLs, look more closely at what those pages return.
- Use Google's tools for the rendered side. The URL Inspection tool in Search Console shows the HTML Google rendered, which you can compare against the raw response.
- Ask the assistants. Give ChatGPT, Claude or Perplexity a specific URL and ask what the page says about prices or features. A vague or wrong answer is a signal, though not proof, since they may use cached data.
Fixes: SSR, SSG, pre-rendering and progressive enhancement
Every fix for JavaScript and AI crawlers aims at the same thing: the content you want cited must be in the first HTML response. There are four common ways to get there.
- Server-side rendering (SSR). The server builds the full HTML on each request, then JavaScript hydrates it for interactivity. Good for pages that change often, such as stock or prices.
- Static site generation (SSG). Pages are built to HTML at deploy time and served as files. Fast, cheap and ideal for articles, guides and service pages. Incremental regeneration lets you rebuild individual pages without a full deploy.
- Pre-rendering. A headless browser renders your existing client-side pages ahead of time and stores the HTML. It is a practical bridge for a legacy single-page app you cannot rewrite soon.
- Progressive enhancement. Write the core content and links in plain HTML, then let JavaScript add filters, animations and widgets on top. If a script fails, the page still says what it needs to say.
Avoid serving a special version only to bots. Google calls dynamic rendering a workaround and not a long-term solution, and recommends server-side rendering, static rendering or hydration instead. Source: Google on dynamic rendering. Serving crawlers content people do not see also risks being treated as cloaking.
Framework notes
Most modern frameworks handle JavaScript and AI crawlers well if you let them produce HTML on the server or at build time. Problems usually come from how they are configured, not from the framework itself.
- Next.js. In the App Router, layouts and pages are Server Components by default, and Client Components are also pre-rendered to HTML on the server. The common trap is fetching core content inside useEffect in a Client Component, which runs only in the browser. Fetch it on the server and pass it down as props.
- Nuxt. Universal rendering is the default: the server sends HTML and the browser hydrates it. Check that nobody set ssr: false globally or on key routes. Route rules let you pre-render routes at build time.
- React SPA (Create React App, Vite with client rendering only). The server sends an almost empty document. Either move to a framework with SSR or SSG, or add pre-rendering for the pages that matter.
- Webflow, Framer, Wix and similar builders. In general, the pages these platforms publish ship their main text in the HTML. The risk sits in additions: embedded widgets, apps from marketplaces, review carousels and content loaded by custom scripts. Test those pieces with view-source, page by page.
What to prioritise first
You rarely need to rebuild a whole site. Start where assistants form their picture of your business.
- Home, about and service or category pages: who you are, what you sell, where, for whom.
- Pricing and product pages, if prices and specifications are loaded client-side.
- FAQ content and comparison pages, which match the questions people put to assistants.
- Internal links and navigation, so crawlers can reach everything else.
Fix one template, retest with the steps above, check the logs again, then move to the next.
Where this fits in technical GEO
Rendering is one part of technical GEO. The first gate is access: if your robots.txt rules for AI bots or your CDN turn crawlers away, rendering never comes into play. The second gate is what we covered here, JavaScript and AI crawlers. The third is making the content easy to interpret, for example with schema markup for AI search, written into the HTML rather than injected later.
If you want the wider context of how assistants retrieve and cite pages, read how AI search works. And if you are not sure where your site stands, rendering checks are part of our free GEO audit.
Frequently asked questions
Do AI crawlers render JavaScript?
Googlebot does, and Gemini and AI Overviews work from that index. In the Vercel and MERJ test of December 2024, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot did not render JavaScript. The vendors do not document this, so it may change.
Is client-side rendering bad for GEO?
Only for the content that matters. Interactive features can stay client-side. Text, prices, links and structured data you want assistants to use should be in the HTML your server returns.
Does my structured data count if a tag manager adds it?
Google can read it after rendering. A crawler that does not run JavaScript will not see it. Writing JSON-LD directly into the page template is the safer choice for the topic of JavaScript and AI crawlers.
Is dynamic rendering still an option?
Google describes it as a workaround, not a long-term solution, and recommends server-side rendering, static rendering or hydration. Pre-rendering the same HTML for everyone is a better bridge.
How often should I retest?
After every redesign, framework upgrade or new third-party widget, and a few times a year otherwise. Crawler behaviour around JavaScript and AI crawlers can change without notice, so check your logs too.


