Is Your React, Vue or Angular Site Invisible to AI - Why and How to Fix It
Why ChatGPT, Claude or Perplexity may not see a JavaScript-rendered site, how to test yours with curl, and which SSR or static generation fixes to apply.
A single-page application (SPA) built with React, Vue or Angular often sends the browser a nearly empty HTML document, then builds the content with JavaScript. A browser handles that fine; a crawler that only downloads the HTML sees an empty shell. And most AI crawlers appear to work exactly that way. This guide reviews what is actually known about their behavior, shows how to check your own site in a few minutes, then walks through the fixes available in Next.js, Nuxt and Astro.
What AI crawlers execute, according to available sources
One important point first: none of the major AI assistant providers officially documents whether its crawlers execute JavaScript. The official pages from OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) and Perplexity (PerplexityBot, Perplexity-User) describe what each crawler does and how to control it, but not their rendering engine. The exception is Common Crawl, whose FAQ states that "JavaScript is not executed" by CCBot, which matters because its archive is widely reused for training. For the assistants, what we know comes from observation.
The most cited reference is an analysis published by Vercel and MERJ in December 2024, based on traffic observed across Vercel's network. Its conclusion: none of the major AI crawlers studied rendered JavaScript. Some download .js files (11.5% of requests for ChatGPT's crawlers, 23.8% for Claude's, according to the study), but they don't run them.
| Crawler | Executes JavaScript? | Source of the information |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI) | Not observed | Vercel/MERJ study, December 2024; nothing in OpenAI's documentation |
| ClaudeBot (Anthropic) | Not observed | Same study; nothing in Anthropic's documentation |
| Claude-SearchBot, Claude-User (Anthropic) | Unknown | No public data found, not documented |
| PerplexityBot (Perplexity) | Not observed | Same study; nothing in Perplexity's documentation |
| meta-externalagent (Meta), Bytespider (ByteDance) | Not observed | Same study |
| CCBot (Common Crawl) | No | Common Crawl FAQ: "JavaScript is not executed" |
| Googlebot | Yes | Google documentation, rendering with headless Chromium |
| Bingbot | Yes | Bing announcement from 2019, rendering with Microsoft Edge |
| Applebot | Yes | Apple's documentation says Applebot "may render the content of your website within a browser"; also observed in the Vercel/MERJ study |
Two caveats. First, the study dates from late 2024, and a crawler's behavior can change without notice. Second, Google's AI features rely on the Google Search index, built by Googlebot, which does execute JavaScript; the Google-Extended token has no separate crawler. Google nonetheless points out that server-side rendering or pre-rendering is still "a great idea" because "not all bots can run JavaScript."
Finally, agents that drive a real browser execute JavaScript like any browser. But that is not what feeds an assistant's search index: to be found and cited, your content should be in the initial HTML.
The typical symptom
Here is what a non-rendering crawler receives from a React application created with a standard project template.
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>My Company</title>
<script type="module" src="/assets/index-4f8a2c.js"></script>
</head>
<body>
<div id="root"></div>
</body>
</html>
The title is there, nothing else is: no text, no internal links, no contact details. A crawler that stops here has nothing to index or cite, and it won't discover your other pages either, since the links only exist after the script runs.
Testing it yourself
1. With curl. curl fetches the HTML without executing JavaScript, just like the crawlers described above. Look for a sentence that appears in the main content of the page.
# Does the raw HTML contain the expected sentence?
curl -s https://www.example.com/pricing | grep -c "Free quote"
# Same test, presenting the user agent of an AI crawler
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
https://www.example.com/pricing | grep -c "Free quote"
# How many internal links are present in the initial HTML?
curl -s https://www.example.com/ | grep -o '<a [^>]*href="/[^"]*"' | wc -l
A result of 0 for a sentence you can see on screen means that text is added by JavaScript. The second test also helps spot a firewall that treats that user agent differently (a 403 status, a challenge page). It is only indicative: it imitates the crawler's name, not its IP address.
2. With view-source:. In Chrome or Firefox, type view-source:https://www.example.com/ in the address bar. You see the HTML as it was received, before any script runs. The Elements panel in DevTools, on the other hand, shows the DOM after scripts have run, so don't confuse the two.
3. By disabling JavaScript. In Chrome, open DevTools, then the Command Menu (Ctrl+Shift+P, or Cmd+Shift+P on macOS), type javascript, choose Disable JavaScript and reload the page. The setting only applies to that tab and only while DevTools stays open. What you see is a good approximation of what a non-rendering crawler reads.
Run these tests on several types of pages (home page, product page, article, contact page), because many sites mix server-rendered pages with parts loaded on the client.
The fixes
| Technique | How it works | When to use it |
|---|---|---|
| Server-side rendering (SSR) | The server produces the complete HTML for each request | Content that changes often or depends on the request |
| Static site generation (SSG) | The HTML is produced once, at build time | Pages that rarely change, such as a business site, blog or documentation |
| Incremental regeneration (ISR, SWR) | Static HTML is cached, then regenerated periodically | Catalogs, content updated several times a day |
| Pre-rendering an SPA | A tool runs the application at build time and saves the HTML for each route | An existing SPA you can't migrate |
| Dynamic rendering | Pre-rendered HTML served to crawlers only | Avoid it; Google calls it a workaround and does not recommend it |
Hydration is not an alternative but the companion of SSR and SSG: the browser receives HTML that is already filled in, then JavaScript "attaches" to that HTML to make it interactive. The crawler reads the HTML; the user gets the interactivity. This is the default behavior of Next.js, Nuxt and Angular with SSR.
The <noscript> tag is only displayed when JavaScript is disabled. It can carry a fallback message or link, but it is not a solution for your main content: you would have to duplicate the entire page, and noscript content that differs from what users see counts as hidden text.
<noscript>
<p>This calculator requires JavaScript. See our <a href="/pricing">price list</a>.</p>
</noscript>
Next.js
With the App Router, pages and layouts are Server Components by default: their content is rendered to HTML on the server or at build time. The classic trap is loading the main content in a client component, after mount.
// app/pricing/page.js: avoid this for content you want indexed
'use client'
import { useEffect, useState } from 'react'
export default function Pricing() {
const [plans, setPlans] = useState(null)
useEffect(() => {
fetch('/api/plans').then((r) => r.json()).then(setPlans)
}, [])
if (!plans) return <p>Loading…</p> // this is all a crawler will see
return <ul>{plans.map((o) => <li key={o.id}>{o.name}</li>)}</ul>
}
// app/pricing/page.js: server-rendered version
export default async function Pricing() {
const res = await fetch('https://api.example.com/plans')
const plans = await res.json()
return <ul>{plans.map((o) => <li key={o.id}>{o.name}</li>)}</ul>
}
Another thing to watch: dynamic(() => import('...'), { ssr: false }) excludes the component from pre-rendering. Reserve it for non-essential elements (an interactive map, a chat widget). For purely static hosting, output: 'export' in next.config.js generates one HTML file per route during next build.
// next.config.js
const nextConfig = { output: 'export' }
module.exports = nextConfig
Nuxt
Nuxt uses universal rendering by default: HTML produced on the server, then hydration. The problem appears when someone has disabled SSR globally with ssr: false, which turns the site into an SPA. routeRules let you choose the rendering mode route by route.
// nuxt.config.ts
export default defineNuxtConfig({
routeRules: {
'/': { prerender: true }, // HTML generated at build time
'/blog/**': { isr: 3600 }, // generated on demand, cached for 1 hour
'/admin/**': { ssr: false }, // client-side rendering only
},
})
For data, useFetch and useAsyncData run during server rendering; a fetch placed in onMounted only runs in the browser. For static hosting, nuxt generate produces one HTML file per page.
Astro
Astro pre-renders every page to static HTML at build time by default, and renders React, Vue or Svelte components to HTML with no client-side JavaScript unless you mark them with a client:* directive. The only trap is client:only, which skips server rendering.
---
import Catalog from '../components/Catalog.jsx'
---
<!-- Rendered to HTML at build time, then hydrated: visible to crawlers -->
<Catalog client:load />
<!-- Rendered only in the browser: invisible without JavaScript -->
<Catalog client:only="react" />
For on-demand rendering, add a server adapter and export const prerender = false to the pages concerned.
Plain React, Vue, Angular
For an existing React SPA, React Router in framework mode offers a prerender option in react-router.config.ts that generates HTML for the listed routes at build time. For Vue, the documented path to server rendering is Nuxt. Angular has built-in SSR: ng new --ssr for a new project, ng add @angular/ssr for an existing one, with a per-route choice of mode (RenderMode.Prerender, RenderMode.Server).
Checklist
- The main text of every page type shows up in
curland inview-source:. - Navigation links are real
<a href>elements present in the initial HTML. - The
<title>, meta description and structured data are in the initial HTML, not injected by a script. - No important page uses
ssr: false,client:onlyor loads its content inuseEffect. - Crawlers don't get a challenge or a 403 from your firewall or CDN (see is Cloudflare blocking AI crawlers), and your robots.txt does not block the JavaScript and CSS files that rendering crawlers such as Googlebot and Applebot need.
To check your site, use the IABIG checker.
Sources
- Vercel and MERJ, The rise of the AI crawler (December 2024)
- OpenAI, Overview of OpenAI Crawlers
- Anthropic, Does Anthropic crawl data from the web
- Perplexity, Perplexity Crawlers
- Common Crawl, FAQ
- Apple, About Applebot
- Google, Understand the JavaScript SEO basics
- Google, Dynamic rendering as a workaround
- Google, Google-Extended in the list of common crawlers
- Bing, The new evergreen Bingbot (2019)
- Chrome DevTools, Disable JavaScript
- Next.js, Server and Client Components
- Next.js, Static Exports
- Next.js, Lazy Loading
- Nuxt, Rendering Modes
- Astro, On-demand rendering
- Astro, Islands architecture
- Astro, Template directives reference
- React Router, Pre-Rendering
- Angular, Server and hybrid rendering
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- llms.txt - Exact Format, Examples and Real Adoption by AI CompaniesThe llms.txt file as specified on llmstxt.org, with examples for a business site and a documentation site, and what AI companies actually confirm about it.
- Is Cloudflare Blocking AI Crawlers on Your Site - How to Check and Fix ItSearch, Agent and Training policies, Bot Preference Sync, AI Crawl Control, pay per crawl, and how to see what Cloudflare blocks and adjust it to your goals.