Our site's nginx access log runs from 6 June to 15 October 2026 (Beijing time): 120,261 lines over roughly 130 days. The site is made up mainly of three Next.js apps. We used it to answer a question that comes up a lot in GEO: when an AI crawler fetches a page, does it take away the HTML the server returns, or the page as it looks after its scripts have run?
The test rests on one very specific signal: requests whose URL carries an _rsc= parameter. Five readings first; every section below gives the method, the identity checks and the sources:
- Among the names on our list, six crawler names sent `_rsc` requests, which means they show signs of running scripts. GoogleOther 839, meta-externalagent 343, OAI-SearchBot 281, Bytespider 233, GPTBot 14 and meta-webindexer 13 (according to Google, GoogleOther is a generic crawler that several product teams may use, not an AI-only one; the other five belong to OpenAI, Meta and ByteDance). All of GoogleOther's, OAI-SearchBot's and GPTBot's
_rsc requests came from inside the address lists their operators publish; all of meta-externalagent's came from Meta's own network (Meta gives no verification method, so this is only supporting evidence); Bytespider could not be verified. - Running scripts does not mean running them at the moment the page is fetched. Most of GoogleOther's, meta-externalagent's and Bytespider's
_rsc requests follow a page fetch within 60 seconds; none of OAI-SearchBot's 281 do, and the median gap to the latest fetch of the page is about 2 hours. - The verifiable names that fetch pages "when a user asks" fetched no script files and sent no `_rsc`. All 173 of ChatGPT-User's requests inside its operator's list are page fetches; Claude-User has only 28 in-list requests (11 pages, 12 robots.txt, 4 sitemap, 1 other), a very small sample; both have 0 script files and 0
_rsc. PerplexityBot and Amazonbot are not of the "when a user asks" kind, and their in-list requests also include 0 script files and 0 _rsc. - Downloading a script file is not the same as running it. ClaudeBot fetched script files 59 times; the 57 that came from inside Anthropic's list all carried no referrer, which looks more like fetching script URLs as ordinary links, and the 2 with an on-site referrer both came from outside the list. Its
_rsc count is 0. - Googlebot, bingbot, Applebot and Baiduspider all have 0 `_rsc`, but that does not show they skip scripts. According to Next.js's front-end code, it recognises these names and does not send them prefetch requests, so the
_rsc signal cannot appear for them in the first place (see point 3 of section 2).
For GEO, the conclusion fits in one sentence: key facts such as your brand name, prices, scope of capabilities and FAQs belong in the HTML the server returns directly, not only in what browser-side scripts render.
A disclosure up front: this log comes from our own site; YinJen, which lives on that site, is our own product, a GEO tool that monitors brand visibility across 12 AI engines. The data covers one site and one framework; it is not an industry average. China's AI engines barely show up in this log, so this article cannot say how they fetch pages (see section 8).
1. Scope: read this section before the numbers
| Item | Value |
|---|
| Data source | The nginx access log of zhimahang.com, the site of Suzhou ZhiMaHang Technology Co., Ltd. (YinJen's article and product pages live on this site) |
| Site | One domain hosting three Next.js apps (App Router), plus a portal home page and a tool page that are static; every _rsc request in this article landed on those three Next.js apps |
| Window | The log's first line is 2026-06-06 23:04 Beijing time; we cut off at 10-15 11:00, and the last line before the cutoff is 10:36; 120,261 lines in total, and the log was not rotated in that period |
| How crawler names are assigned | By the name each request declares in its User-Agent header. The name list is one we drew up in advance (common AI crawlers, search engine spiders and the like, plus the Doubao app); User-Agents outside the list are not put in any group (see point 1 under section 3). Matching is case-insensitive, in a fixed order, first match wins, one name per line (for example, a line carrying both Amazonbot and Amzn-SearchBot counts as Amzn-SearchBot). Names can be spoofed, so every name gets a separate identity check, see section 4 |
_rsc request | A request whose query string contains an _rsc= parameter |
| Script file | A request whose path is under /_next/static/ and ends in .js |
| Page fetch | A request that is not for a script, stylesheet, image, font or other static asset, and not for robots.txt, a sitemap or _rsc; a "successful page fetch" counts only 2xx responses |
| Referrer | The Referer request header. When a browser loads a page's scripts or sends a prefetch request, it includes the current page's address |
| When pulled | Log: the main tables at 11:55 to 12:12 Beijing time on 2026-10-15, plus two follow-up passes at 12:48 to 12:54 and 13:27 to 13:28, all aggregated read-only in the server's memory over ssh, outputting counts only and writing no files on the server; the operators' address lists used for checks were downloaded at 11:51 to 11:53. Page checks: raw responses of our own pages fetched with curl at 11:59 to 12:17 (used in sections 2 and 7), and the product site's pricing page fetched again at 14:11 on 10-16 (used in point 4 of section 7), all outside the log window |
| What never appears | Any visitor's IP address, or any full hostname from a reverse lookup |
| Time zone | The log itself is in Beijing time (+0800), and all dates in this article are Beijing time |
2. Why _rsc works as a signal and script files do not
The Next.js App Router has a default behaviour: once a page is running in a browser, it fetches ahead "the data for the next page" for internal links visible in the viewport. That request's URL ends in ?_rsc= plus a value computed by front-end code. Three points first:
- The HTML contains no such addresses. We checked the raw responses of 7 pages (the product site's home page, the article list, two articles, the pricing page and the home pages of the other two apps) and two sitemaps one by one; the characters
_rsc do not appear once. To send such a request, someone has to run the page's scripts, or get the address from somewhere that did. - Script file addresses are written right into the HTML. Each of those 7 pages references 9 to 10 script files under
/_next/static/. A crawler can fetch them as ordinary links without executing anything. So "fetched a script file" only says it downloaded the file, not that it ran it. The referrer helps tell the two apart: a browser loading a page's scripts sends the page address, while fetching a script as a standalone link usually does not. - Next.js does not prefetch for some names. Its front-end routing code has a "do not prefetch for bots" check (one app runs version 15.5.18 and the other two run 15.5.20, and this check and the list below are the same in both versions). The list includes Googlebot, names containing "Google-" or "-Google", bingbot, Applebot, Baiduspider, facebookexternalhit and others. Going by this code, a crawler whose name is on that list will not send
_rsc even if it runs every script (assuming the User-Agent the scripts read during rendering is the same as the one in the request header, which the log cannot show). GPTBot, OAI-SearchBot, ClaudeBot, GoogleOther, Bytespider and meta-externalagent are not on the list.
So the groups below show whether the signal is present, not the whole answer to whether scripts were run.
3. Three groups by signal (counted by self-declared User-Agent)
Group A: sent `_rsc`, showing signs of running scripts
| Name | Total requests | Successful page fetches | Script files | _rsc | First to last _rsc |
|---|
| GoogleOther | 1,410 | 268 | 170 | 839 | 06-29 to 10-01 |
| meta-externalagent | 986 | 127 | 53 | 343 | 10-03 to 10-12 |
| OAI-SearchBot | 1,453 | 108 | 21 | 281 | 07-15 to 10-09 |
| Bytespider | 933 | 59 | 256 | 233 | 08-26 to 10-14 |
| GPTBot | 1,206 | 236 | 133 | 14 | 08-12 to 10-11 |
| meta-webindexer | 71 | 5 | 16 | 13 | 09-05 to 09-27 |
Group B: names not on that Next.js list, with 0 `_rsc`
| Name | Total requests | Successful page fetches | Script files | _rsc |
|---|
| ClaudeBot | 3,145 | 146 | 59 (2 with an on-site referrer) | 0 |
| ChatGPT-User | 510 | 208 | 0 | 0 |
| Claude-User | 445 | 52 | 0 | 0 |
| Perplexity-User | 283 | 2 | 0 | 0 |
| PerplexityBot | 269 | 58 | 1 | 0 |
| Amazonbot | 265 | 81 | 0 | 0 |
Group C: names on that Next.js list, for which the `_rsc` signal does not apply
| Name | Total requests | Successful page fetches | Script files | _rsc |
|---|
| Googlebot | 4,075 | 2,069 | 747 | 0 |
| Google-InspectionTool | 2,117 | 280 | 906 | 0 |
| bingbot | 1,754 | 885 | 63 | 0 |
| Baiduspider | 784 | 246 | 212 | 0 |
| Applebot | 518 | 65 | 220 | 0 |
How to read this:
- All 1,723 `_rsc` requests in Group A carry the address of a page on our site as the referrer, which is what a request sent from inside a page by a browser looks like. For reference, browser-like User-Agents (a rough filter: starting with Mozilla/ and containing none of bot, spider, crawl and similar words) sent 4,385
_rsc requests in the same window, all with an on-site referrer as well. In the whole window there are 6,314 _rsc requests: besides these 1,723 in Group A and the 4,385 from browser-like User-Agents, 106 come from the Doubao app's in-app browser (see section 8) and 100 from outside our name list. Of those 100, 5 declare YisouSpider (not verified, and all with an on-site referrer too), and 95 come from a few User-Agents that are neither on our list nor browser-like (8 of them are the headless browser HeadlessChrome); they are not put in any group in this article. - Every script fetch in Group C carries an on-site referrer (Googlebot 747, Google-InspectionTool 906, bingbot 63, Baiduspider 212, Applebot 220), which is what loading a page's scripts looks like. Google's documentation says it processes JavaScript pages in three phases: crawling, rendering and indexing. The log alone cannot prove execution, though. Google-InspectionTool is the name used by tools such as Search Console's URL Inspection and the Rich Results Test, usually triggered by site owners clicking a test in those tools; we use them often ourselves. Group C is split by name: checking each line's raw User-Agent against those two Next.js checks, 169 lines under the Googlebot name (all Googlebot-Image/1.0) do not match, and those lines have 0
_rsc as well. - In Group B, ClaudeBot fetched script files 59 times, and 57 had no referrer, as if it lifted script URLs out of the HTML and fetched them separately. ChatGPT-User, Claude-User, PerplexityBot and Amazonbot fetch no script files and send no
_rsc (one script request under the PerplexityBot name falls outside its list); their in-list requests are for pages, robots.txt and sitemaps, plus 11 images and 1 text file, and the four names have another 271 requests for files with static extensions, all from outside the lists. Not one of the 283 requests under the Perplexity-User name passed the identity check (see section 4), so we draw no conclusion from it. - All three tables count by self-declared User-Agent, so they mix in spoofed names and our own requests. For example, 102 of GoogleOther's 268 successful page fetches were sent by us on 9 and 10 October (see section 8); 1,217 of Googlebot's 2,069 successful page fetches are not on Google's list, and at least 414 of those were sent by us on those two days. The figures after verification are in section 4.
4. The names are self-declared: what is left after checking
We used the verification methods each operator publishes: where there is an address list, we checked whether the address is on it; where there is a reverse-lookup rule, we checked that the reverse-lookup hostname belongs to the operator's domain and that a forward lookup returns the original address. Where both are offered, passing either counts.
| Name | Verification method | Passed, of all requests | Passed, of _rsc | The part that did not pass |
|---|
| GoogleOther | Google's published address list for its common crawlers | 1,298 / 1,410 | 839 / 839 | 112 requests, 111 of them page fetches (102 are our own requests, see section 8); no scripts, no _rsc |
| OAI-SearchBot | OpenAI's published OAI-SearchBot address list | 1,150 / 1,453 | 281 / 281 | 303 requests, including 205 page fetches, 94 other static assets and 1 script; 0 _rsc |
| GPTBot | OpenAI's published GPTBot address list | 835 / 1,206 | 14 / 14 | 371 requests, including 289 page fetches, 79 other static assets and 1 script; 0 _rsc |
| meta-externalagent | Meta's documentation gives no verification method; supporting evidence: whether the address falls inside the networks of AS32934, Meta's own autonomous system | 703 / 986 | 343 / 343 | 283 requests, including 206 page fetches and 75 other static assets; 0 scripts, 0 _rsc |
| Bytespider | The operator's documentation sits behind a login page and could not be read, so there was no way to verify | — | — | Supporting evidence left incomplete: for the addresses behind the 233 _rsc and 256 script requests, reverse-lookup hostnames are all under bytedance.com, but forward lookups through the server's resolver did not return the original addresses |
| ChatGPT-User | OpenAI list | 173 / 510 | — | All 173 in-list requests are page fetches; 0 scripts |
| Claude-User | Anthropic list | 28 / 445 | — | 0 scripts in the list |
| PerplexityBot | Perplexity list | 71 / 269 | — | 0 scripts in the list |
| Perplexity-User | Perplexity list | 0 / 283 | — | Not one is on the list |
| Amazonbot | Amazon list | 103 / 265 | — | 0 scripts in the list |
| ClaudeBot | Anthropic list | 2,931 / 3,145 | — | 57 script fetches in the list |
| Googlebot | Google list | 2,522 / 4,075 | — | All 747 script requests are in the list |
| Google-InspectionTool | Google list | 2,007 / 2,117 | — | All 906 script requests are in the list; the 110 that did not pass are all our own requests (see section 8) |
| bingbot | Microsoft list | 1,475 / 1,754 | — | 61 / 63 scripts in the list |
| Baiduspider | Reverse plus forward lookup, per Baidu's documentation | 463 / 784 | — | 210 / 212 scripts passed |
| Applebot | Apple list, or reverse plus forward lookup | 364 / 518 | — | 220 / 220 scripts passed |
How to read this:
- For the three verifiable names in Group A, every `_rsc` request came from inside the operator's list. meta-externalagent's 343 and meta-webindexer's 13 also all fall inside Meta's networks; we checked against two public routing datasets separately, and both gave the same result.
- The part that failed the checks hardly fetches scripts and never sends `_rsc`. It includes our own test requests (all of the 938 from October described in section 8 are in it), and may also include programs spoofing these names and addresses the lists do not cover; in this window it essentially fetched only pages and static assets. So without checking identity first, a conclusion like "crawler X only sees HTML" gets skewed by this traffic.
- Verification results are a snapshot. Address lists and reverse-lookup records change; this article uses the versions current on the day of the pull.
5. Are scripts run at the moment the page is fetched?
For each _rsc request we took its referrer and looked back for when that name fetched the page:
| Name | _rsc | Within 60 s after the same-named crawler fetched the referring page | Same address for page fetch and _rsc? | Gap to the latest fetch of the page |
|---|
| GoogleOther | 839 | 689 (82.1%) | Same address 281 times | 821 within 600 s |
| meta-externalagent | 343 | 343 | Same address 3 times | All within 60 s |
| Bytespider | 233 | 233 | Same address 233 times | All within 60 s |
| OAI-SearchBot | 281 | 0 | — | Shortest about 3 minutes; quartiles about 31 minutes, 2 hours and 3.8 hours; longest about 8.8 hours |
| GPTBot | 14 | 0 | — | About 32 minutes to 4 hours |
In the GoogleOther and meta-externalagent rows, "fetched the referring page" counts only requests from inside the list or inside Meta's networks; in the OAI-SearchBot and GPTBot rows, "the latest fetch of the page" counts fetches from any address on OpenAI's three lists (GPTBot, OAI-SearchBot, ChatGPT-User). "The page" means the full URL, so the English version with ?lang=en counts as a different page.
How to read this:
- GoogleOther, meta-externalagent and Bytespider look like they render as they fetch. 791 of GoogleOther's
_rsc requests fall within 5 seconds (5 seconds included) of another _rsc from the same address with the same referring page, arriving in bursts, as if a page that has just started running prefetches the links in its viewport together. meta-externalagent usually fetches the HTML and sends _rsc from different addresses: one place fetches the page, another renders it. All 233 of Bytespider's come from one and the same article page, spread over 33 days, one round per day. - OAI-SearchBot looks like it takes the page away first and renders it some time later. The referring pages of all 281
_rsc requests had been fetched beforehand by addresses on OpenAI's lists, but never within 60 seconds, and only 26 within 600 seconds. For 262 of the 281, the same address sent nothing except other _rsc requests in the 60 seconds before or after; across the whole window it fetched script files only 21 times. This is an inference the log cannot prove: it looks as if the HTML is fetched first and rendered elsewhere later, perhaps with cached scripts. Google's documentation also says pages go into a rendering queue, where they may wait a few seconds or longer. - What this means for GEO: the "page after its scripts ran" that a crawler sees may be rendered from HTML fetched hours earlier. After you change a page, different crawlers see the new version at different times.
6. By month: the same name's behaviour changes too
_rsc requests per month by name (June counted from the 6th, October up to the 15th):
| Name | June | July | August | September | October |
|---|
| GoogleOther | 291 | 223 | 225 | 89 | 11 |
| meta-externalagent | 0 | 0 | 0 | 0 | 343 |
| OAI-SearchBot | 0 | 5 | 15 | 179 | 82 |
| Bytespider | 0 | 0 | 8 | 127 | 98 |
| GPTBot | 0 | 0 | 4 | 4 | 6 |
| meta-webindexer | 0 | 0 | 0 | 13 | 0 |
- meta-externalagent made 349 requests from June to September without a single
_rsc; they first appeared on 3 October, and it sent them on 4 of the 13 days from then to the end of the window. GoogleOther went the other way: none after 1 October, and its total requests fell from 395 in July to 188 in September. - The site kept adding articles and pages over this period, so the monthly denominators differ and this table does not show a trend. It only shows that "does crawler X run scripts" has no fixed answer, only an observation within a given window.
7. What this is good for in GEO
What you can use it for:
- Put key facts in the HTML the server returns directly. The verifiable ChatGPT-User, Claude-User, PerplexityBot and Amazonbot in this article fetched no script files and sent no
_rsc, and the crawlers that do run scripts may do so hours later. We checked our own pages: we took every run of 8 or more consecutive Chinese characters in the body of two articles and searched for its first 8 characters, and 44 / 44 and 277 / 277 appear in the visible text of the HTML the server returns; the plan prices on the pricing page are in the HTML too, with the annual price in the visible text and a monthly price range in the structured data. The check is simple: from the command line, fetch the raw HTML once with the User-Agent of each crawler you care about (the same page can come back differently depending on the User-Agent, see point 4), and search it for the sentences and numbers you care about most; whatever you cannot find, a crawler that only takes HTML cannot see. These requests also end up in your own log, so remove them or state them before you count crawlers (section 8 of this article is a case in point). - Look in your own logs for requests that only a running script would send. On a Next.js App Router site, count
_rsc; on other frameworks, look for data API requests that only front-end code sends. Do not treat "fetched a script file" as proof of execution; look at the referrer. - Verify identity before drawing conclusions. In this article, 371 of the 1,206 requests under the GPTBot name are not on OpenAI's list, and not one of the 283 under the Perplexity-User name is on its list. Counting by name alone mixes in traffic that fails verification (spoofed names, our own test requests and so on).
- If you use Next.js 15.2 or later, check where your metadata ends up. From 15.2 on, Next.js by default streams metadata such as the title and description, and only for crawlers on its list does it deliver it all in the
<head> at once; the default list contains no AI crawler names such as GPTBot or ClaudeBot. In July we added the common AI crawler names to the htmlLimitedBots setting. Note that once you set this option, Next.js's built-in default list stops applying altogether, so you have to write the default list in as well; we wrote the two lists in together. If you list only AI crawlers, then bingbot, Baiduspider, Applebot and the others on the default list no longer get the metadata delivered all at once. After that change, when we fetched the pricing page on 15 October with a User-Agent that is not on the list, the title and description appeared in the <body>; on article pages they are in the <head>. On 16 October, on the product site, we added 20 more names that had appeared in our log but were missing from the list, including Claude-User and meta-webindexer (the other two apps still use the July list). When we fetched the product site's pricing page that day with a User-Agent carrying Claude-User, the title and description were in the <head>; with a User-Agent not on the list they were still in the <body>. This configuration change came after our data cutoff, and the tables above count only log entries from before it, so their numbers are unaffected. This setting only governs how the server outputs metadata and does not change the front-end prefetch check described in section 2: GPTBot, OAI-SearchBot, Bytespider and meta-externalagent are all on our list and still sent _rsc. The list has to keep up with the log.
What you cannot use it for:
- This article makes no promise about indexing, rankings or any other outcome.
Running scripts does not mean a page will be cited by AI, and taking only the HTML does not mean it will not be.
- It does not generalise to other sites or frameworks. This is one site, the Next.js App Router, and 130 days of logs.
- 0 `_rsc` does not mean scripts were not run. For Group C, Next.js's front-end code does not send it; Group B may run scripts in ways we cannot see in the log.
- It cannot tell you which crawler is better. A count is only a count.
8. Limits of this data
- `_rsc` is just one signal. The Next.js version, prefetch settings on links and the viewport size during rendering all affect whether, and how often, it is sent. Nor can we rule out a crawler getting an
_rsc address from elsewhere (for example from another system that did run the scripts) and then fetching it: the gap between page fetches and _rsc for OAI-SearchBot and GPTBot leaves that possibility open. - Verification is a snapshot, and the Meta row is only supporting evidence. Meta gives no verification method, so we used its own networks; Bytespider's operator documentation could not be read, so its check was not completed.
- "Browser-like User-Agent" is a rough filter. It serves only as a reference and feeds no conclusion.
- The log records only the request itself. The dedicated request headers that prefetch requests carry are not recorded in the access log, so we can only look at
_rsc= in the URL. - We sent requests ourselves. In September we scanned our own site with curl using the names of Googlebot and other search engines; on 9 and 10 October, to find out where our metadata ends up, we again used curl with the names Googlebot, GoogleOther, Google-InspectionTool, bingbot and Baiduspider to fetch the pages of the three apps one by one, 938 requests over those two days (Googlebot 414, bingbot 205, Google-InspectionTool 110, Baiduspider 107, GoogleOther 102), all page fetches, identified by address, time and the exact User-Agent strings in our scripts. None of these requests passes the identity checks, so they do not affect the "passed" column in section 4; but they are in the tables counted by self-declared name in section 3, and GoogleOther's 268 successful page fetches include these 102. We did not remove them from the tables; we state them here instead. The page checks we ran on 15 October happened after the cutoff and are outside the window.
- China's AI engines barely show up in this log. Requests whose User-Agent contains kimi (114 lines), deepseek (71), qwen (57), hunyuan (42) or chatglm (42) are few (one line can carry several names); the only successful page fetches are 7 under deepseek, all have 0
_rsc, and we did not verify these names. The 222 lines under Doubao all carry the User-Agent of the in-app browser of the Doubao mobile app, not a crawler, and their 106 _rsc requests are not counted as crawler requests in this article. This article cannot say how China's engines fetch pages. - Snapshots change. The log keeps growing and the address lists get updated; every number in this article is a reading at the time given above.
Sources and scope: The log figures come from the nginx access log of our site zhimahang.com (2026-06-06 23:04 to 2026-10-15 11:00 Beijing time, 120,261 lines), aggregated read-only in three stretches between 11:55 and 13:28 on 15 October (times in the "When pulled" row of the scope table); names are as self-declared in the User-Agent, identities were checked against the address lists or reverse-lookup rules published by OpenAI, Google, Anthropic, Perplexity, Amazon, Microsoft, Apple and Baidu, Meta was checked against AS32934's networks as supporting evidence, and Bytespider could not be verified. The page checks in sections 2 and 7 (whether _rsc appears in 7 pages and two sitemaps, how many script files each page references, the prices and metadata position on the pricing page, and the 44 / 44 and 277 / 277 found by searching for the two articles' body text from our content database) come from fetching our own pages with curl at 11:59 to 12:17 Beijing time on 15 October, outside the log window; the Next.js versions come from the three apps' dependency lock files, the prefetch check and the default list from the installed Next.js source code, "from 15.2 on" and "setting the option drops the default list" from Next.js's own documentation, and "added to htmlLimitedBots in July" from our own commit history; "added 20 more names on the product site on 16 October" and "the other two apps still use the July list" come from our commit history and deployment records, and the metadata position on the product site's pricing page after that change from fetching our own pages with curl at 14:11 Beijing time on 16 October, also outside the log window. Measurements on a single site and a single framework, not an industry average.
This article was written by Suzhou ZhiMaHang Technology Co., Ltd. (the YinJen team). The access-log details used in this article have not been published (the log contains visitors' addresses); our public citation datasets are under the GitHub organization ZhiMaHang (verified domain ownership: zhimahang.com). Website: zhimahang.com
Data in this article comes from a read-only count and check of the author's company's own website (log side: the nginx access log from 2026-06-06 to 10-15, Beijing time; page side: curl requests to our own pages on 10-15 and 10-16; framework versions from our own code repositories). The text was drafted with AI assistance and reviewed by the author before publication.
Further reading
