JavaScript links can make your pages invisible to AI search

“Google renders JavaScript” has been the standard answer to client-side rendering concerns for years, and it got easier to believe in 2019, when Googlebot went evergreen, continuously running on the latest version of Chromium.

The increasing flow of search traffic through AI assistants renders that wisdom incomplete because the crawlers for AI assistants behave nothing like Googlebot.

To get numbers rather than assumptions about what that means for JavaScript links, I ran a controlled experiment. I set up one site with half its internal links hard-coded in HTML and the other half injected by JavaScript (JS). I added a logging layer that recorded every bot request for 41 days.

The results were more lopsided than I expected.

• Google’s crawler stack was the only one that executed JavaScript and followed the injected links.
• Almost none of that work came from Googlebot, the crawler that actually builds the search index.
• Googlebot reached only 2% of the JavaScript-linked pages.
• Meanwhile, GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta’s and Amazon’s crawlers, and every other bot I tracked found exactly zero.

How the experiment worked

The test site is a Brazilian business classification directory with roughly 2,400 pages, a project I own and operate. The experiment ran on the site’s hierarchy: 21 top-level sections branching into divisions, groups, and classes, 1,062 pages in total. The remaining pages are flat leaf pages that sat outside the test.

I split the 21 sections into two groups, alternating to avoid clustering bias:

• HTML group (11 sections): Every child link was hard-coded in the raw HTML so that any bot can parse and follow them.
• JS group (10 sections): Every child link, at every level of the hierarchy, was injected by JavaScript after page load. The raw HTML contained an empty div, and a bot that doesn’t execute JavaScript would see no links at all.

To keep the test clean, I removed every alternative discovery path:

• Sitemaps: Disabled so that all sitemap URLs returned 404.
• Breadcrumbs: Removed so deep pages couldn’t leak links back up the hierarchy.
• Internal hierarchy panels: Removed for the same reason.

The only way to reach a division page in the JS group was to render JavaScript on its parent section page.

Site hierarchy split into 11 HTML-linked sections and 10 JavaScript-linked sections, with all discovery paths below JS sections requiring JavaScript execution.

Middleware logged every bot request to a database, including details on path, user agent, referrer, IP, and timestamp. Googlebot and Bingbot IPs were verified via reverse DNS to filter out spoofed user agents. Each request was tagged as belonging to the HTML group or the JS group based on its URL path.

One measurement detail matters. Section pages for both groups were always linked from the index in plain HTML. A bot hitting a JS-group section page proves nothing.

The experiment signal starts one level deeper: A hit on a division page in the JS group is only possible if the bot rendered the parent page’s JavaScript and followed the injected link.

Phase 1: Crawlers arrive as the test begins

Crawlers arrived fast and crawled hard for the first 72 hours.

GPTBot swept the site within hours of deployment and crawled all 759 pages of the HTML group on day one, all the way down to class level. In the JS group, it hit all 10 section pages and stopped. It never reached a single division page.

ClaudeBot followed the same pattern a day later: deep, systematic HTML crawling and a hard stop at the JavaScript boundary.

Google’s two crawlers behaved differently from each other. GoogleOther reached its first JS-injected division page 54 minutes after launch. Googlebot took 9.9 hours. Both worked progressively deeper into the JS hierarchy over the following days, which is only possible if they rendered the parent pages and followed the injected links. Google documents rendering for Googlebot. For GoogleOther, this is an observation from my logs, not documented behavior.

The crawl burst lasted three days. On day four, discovery dropped about 90% for both groups, and the coverage numbers froze.

Daily page discovery by crawler showing GoogleOther’s three-day burst, GPTBot’s single-day sweep, and Bingbot starting on day 10.

Only Google executed the JavaScript

The full scorecard at the end of Phase 1, after 27 days, displays counts covering hierarchy pages only (division level and deeper), where a JS-group hit is unambiguous proof of JavaScript execution. 

One sizing note: The HTML sections hold 748 hierarchy pages and the JS sections 293, so for Google, the table also shows coverage as a share of each group:

Even Google barely touched the JavaScript pages

The two Google rows are doing very different jobs, and the difference matters more than the raw counts. Google’s crawler stack, GoogleOther, passed the rendering test. Googlebot, the crawler that feeds Search, mostly sat it out.

This means the 66% and 48% on the GoogleOther row say nothing about whether those pages can appear in Google. The row that speaks to search visibility is Googlebot: 5% of the HTML group and 2% of the JS group.

Coverage varied by agent

The two groups differ in size, so the fair comparison is the share of each group’s own pages that got crawled.

Googlebot reached 5% of the 748 HTML-linked hierarchy pages and 2% of the 293 JS-linked ones. Six of the 10 JS sections got no Googlebot hierarchy crawl at all in 27 days. GoogleOther is where the volume was: 66% of the HTML group and 48% of the JS group, with its JS coverage falling by depth, from 67% of divisions to 42% of classes.

Those are useful numbers for understanding how rendering behaves, and they aren’t search visibility. For Search, the relevant figure is 2%.

One caveat on that gap. GoogleOther out-crawled Googlebot roughly 14:1 on HTML pages here. I read that as a crawl-budget artifact of a brand-new domain with no external links, not as evidence that GoogleOther is the more thorough crawler. On an established site, I would expect Googlebot’s share to be much higher.

What the split does show is that a site owner watching raw crawl logs can badly overestimate their search coverage if they count every Google user agent as one crawler.

JavaScript was discovered later than HTML 

GoogleOther accounts for nearly all of Google’s crawling here, so its timings are the only ones with enough volume to measure. Its JS-linked pages were discovered 18% to 27% later than its HTML-linked pages at every depth. 

The lag compounds: Google must fetch the parent, queue it for rendering, execute the script, extract the URLs, and then schedule the children. HTML links skip the middle steps.

Distribution was uneven

GoogleOther’s JS coverage wasn’t spread thin evenly. One JS section got 52 pages crawled while two others sat at one page each for weeks. Rendering appears to be a queue with its own budget, and some branches never came up.

Coverage of each link group over time for both Google crawlers, with GoogleOther flattening at 66% for HTML-linked pages and 48% for JavaScript-linked pages while Googlebot stays near the floor at 5% and 2%.

One result ran against my expectations: Once GoogleOther discovered a JS-linked page, it recrawled it as fast as comparable HTML pages, and at two of the three depth levels 24% to 27% faster. The JavaScript delay is entirely front-loaded in discovery.

Once a page gets past that stage, it’s treated normally, sometimes even better than normal. The pages that never get past it are the ones that hurt. They aren’t ranking slowly. They just aren’t there at all.

All other crawlers ignored the JavaScript group

My middleware initially lumped Meta-ExternalAgent and Amazonbot into a generic “unknown bot” bucket, but splitting the logs by user agent revealed them as two of the heaviest crawlers on the site, with roughly 2,000 hits each. 

Meta-ExternalAgent is the crawler Meta uses to gather training data for its AI models. Amazonbot feeds Amazon’s search and AI products. Both crawled the HTML group deeply, hit the 10 JS-group section pages, and never went one level further.

HTML versus JavaScript page discovery per crawler, with only Googlebot and GoogleOther showing any JavaScript-linked pages.

The zeros held for the entire phase. Bingbot was active on 20 of the 27 days of Phase 1, recrawled the 10 JS-group section pages on 18 of those days, and never followed one injected link.

GPTBot crawled the entire HTML group and 0% of the JS hierarchy.

PerplexityBot stayed on the homepage and robots.txt for all of Phase 1. It reached its first hierarchy page on day 40, well after the recovery portion of the test began.

Something odd happened with Bingbot

Bingbot did reach three JS-group hierarchy pages on days 26 and 27, after three weeks of daily recrawls that never went past the section level. However, the crawl pattern points to discovery through another path rather than JavaScript execution, since Bingbot’s coverage of those sections never expanded beyond those isolated URLs.

Microsoft has documented an evergreen, Edge-based Bingbot that can render JavaScript since 2019, but across 20 days of active crawling on this site, it never followed a single injected link, so whatever rendering capacity exists wasn’t applied here.

User requests from AI could not reach JavaScript links

ChatGPT-User deserves attention because it represents real user demand. This is the agent that fetches pages when someone asks ChatGPT a question.

During the experiment, real user queries pulled it to dozens of HTML-linked pages. Pages behind JavaScript links were unreachable for those same queries. So the pages a real person was actively trying to reach were exactly the ones it couldn’t get to.

Crawl logs don’t reveal indexing

Everything above is discovery, not indexing. A page can be crawled and still not make it into the index, and indexing determines whether the page shows up in Search at all. 

Server logs are the right tool for proving what a crawler could reach, which is the question this experiment was built to answer, but they’re the wrong tool for proving visibility. Search Console is the arbiter there.

Phase 2: Not all crawlers passed the recovery test

After 27 days, I converted all JavaScript links to hard-coded HTML and watched what happened. Nothing about the pages themselves changed here. The URLs, the content, and the server were identical to Phase 1. The one thing I touched was the links, which now sat in the HTML instead of being injected by JavaScript.

Some crawlers were quick to catch up

GPTBot responded within two days. It swept every formerly blocked section in a single sweep lasting under three hours and picked up 250 pages it had never crawled before, against zero in 27 days of Phase 1.

Bingbot followed with a systematic sweep, adding 212 pages across the same sections it had ignored for weeks, reaching class level in sections where it had never gone past the entrance.

Meta-ExternalAgent joined by day 41 with a sweep of its own.

Others were not

Amazonbot had finished its crawl and left the site on day 21, six days before the conversion, so its JS-group coverage ended at zero.

ClaudeBot stayed at zero in a different way: It kept recrawling the site after the fix, hundreds of hits through day 41, but only on pages it already knew. It never picked up a single newly linked page. Whatever internal map ClaudeBot drew on its first pass, it seemed to keep working from that same map for weeks, even as it carried on hitting the site.

Cumulative page discovery before and after the conversion to HTML links after 27 days, showing immediate recovery by GPTBot and Bingbot, a flat GoogleOther line, and a Googlebot line at the floor.

The slowest responder, ironically, was Google. Its aggressive first-visit budget was already spent. By day 41, Googlebot had visited exactly one page in the converted sections, and GoogleOther had visited none.

Fixing link architecture after launch recovered AI crawler coverage quickly, but with Google, that first crawl of a brand-new site looks a lot harder to win back a second time.

What I’d tell a client

Iceberg-style diagram showing every HTML-linked page found by at least one crawler at every depth, against GoogleOther coverage of JavaScript-linked pages falling from 67% to 42% as depth increases and Googlebot coverage at 5% or below.

This is one site, one niche, one 41-day window, and a fresh domain with no external links pointing at the buried pages. Established sites with strong signals may see Googlebot invest more in rendering. 

The non-Google results leave less room for interpretation: Eight crawlers failed to execute the JavaScript and follow the injected links across every crawl burst and recrawl cycle.

My takeaways:

• Audit what your raw HTML actually contains. Disable JavaScript and click through your critical paths, or compare the raw source against the rendered DOM. If navigation to revenue pages only exists after hydration, every AI crawler is blind to it.
• Treat “Google renders JS” as the exception, and a partial one. Rendering happened, but the crawler that feeds Search reached 2% of my JS-linked pages. Check which Google user agent your logs are actually showing you before you count that coverage as visibility.
• Prioritize server-side rendering for links above content. A widget that renders in JavaScript just makes a page slightly worse. Navigation that renders in JavaScript is a different problem because it can drop whole sections of your site out of every index that feeds an AI assistant.
• Fix it sooner rather than later. GPTBot re-swept within 48 hours of my Phase 2 conversion, and Bingbot within a week. Google took weeks and still hadn’t returned to the converted sections by day 41.

The rendering question used to be about whether Google could see your content. With AI assistants becoming a real referral channel and the web reshaping itself around agents, the question is now whether other crawlers can. If your links live in JavaScript, they can’t.