llms.txt is a Markdown index placed at a site's root path and written for large language models, and "should we publish one?" is a question that comes up a lot in GEO. We put one on our own site on 8 July 2026, which makes a full 100 days by 15 October; the nginx access log for those 100 days has 104,645 lines, and this article answers one thing only: do AI crawlers actually come and fetch it?
Here are five readings; how each number was computed, how identities were checked and where the data comes from are set out in the sections that follow:
- Verifiable AI crawlers fetched the llms files 0 times in 100 days. The llms files on the site can be reached at five addresses (the llms.txt at the root and under the three apps' paths, plus the product site's llms-full.txt). In this period there were 210 requests for llms files: 196 to those five addresses, plus 14 for an llms-full.txt at the root (there is no file there, and it returns 404); not one came from an AI crawler whose identity could be verified.
- The same crawlers fetched robots.txt and sitemaps 3,791 times. Among the verified requests from 7 names run by OpenAI, Anthropic, Perplexity and Amazon, there were 1,222 robots.txt requests and 2,569 sitemap requests; OAI-SearchBot fetched robots.txt on every one of the 100 days, and GPTBot fetched sitemaps every day. They fetch pages too: 666 successful page fetches.
- All 23 llms-file requests that declared the name of an AI crawler or search-engine spider were ours. 22 were our own site checks using curl under Googlebot's name, and 1 was the Claude-User left behind when we fetched the file with Claude Code, Anthropic's command-line coding tool.
- Of the 210 requests, at least 124 can be confirmed as ours. 17 of them were sent by the website diagnosis in our own product (YinJen), with a User-Agent that reads like an ordinary browser. The other 86 of the 210 are command-line and script requests, three third-party website data collectors, and requests whose User-Agent looks like an ordinary browser and which we cannot identify (see section 2).
- Even going back to the start of the log (6 June), a verified AI crawler touched an llms file only once. On 7 July, before the file went live, a verified Claude-User requested
/llms.txt at the root and got a 404.
For GEO: an llms.txt is fine to publish, and cheap; but do not keep key facts such as your brand name, operating entity and scope of capabilities only in it. What AI crawlers actually fetch is robots.txt, sitemaps and the pages themselves, so key facts belong in the HTML the server returns directly.
First, our interest: YinJen is our own product, a GEO tool that monitors brand visibility across 12 AI engines, and its article and product pages live on this site, so this is our own site's log. Readings from one site are not an industry average. Our previous article used the same log (it looked at whether crawlers run scripts after fetching a page); this one is mainly about machine-readable files such as llms.txt, and a few figures in section 3 are the same as in that article (Claude-User's 12 robots.txt requests and the 283 requests under Perplexity-User).
1. Scope
| Item | Value |
|---|
| Data source | The nginx access log of zhimahang.com: the site belongs to Suzhou ZhiMaHang Technology Co., Ltd., and YinJen's article and product pages are hosted on it |
| Window | From 2026-07-08 00:40:51 Beijing time (when the product site's llms.txt first returned 200) to 10-16 00:00 (exclusive); the last line in the window is at 10-15 23:55; by date that is 07-08 to 10-15, 100 days, 104,645 lines, and the log was not rotated in that period |
| llms files | Requests whose path ends in /llms.txt or /llms-full.txt, whatever status code they got. Five addresses can be reached: /llms.txt at the root (passed by the gateway to the product site's copy, byte-for-byte identical), llms.txt and llms-full.txt under the product site's path, and one llms.txt under each of the other two apps' paths; /llms-full.txt at the root has no file and returns 404 |
| robots.txt, sitemap | Requests whose path ends in /robots.txt, or whose file name starts with sitemap and ends in .xml; counted by request, including 301 redirects and 304 "not modified" responses |
| Successful page fetch | A page request that returned 2xx; static assets such as scripts, stylesheets, images and fonts, robots.txt, sitemaps, llms files and _rsc prefetches do not count as pages |
| How crawler names are assigned | By the name written in the User-Agent request header. We drew up a name list in advance: 34 names of AI crawlers, search engine spiders and the like, plus the Doubao app's in-app browser; matching ignores case and goes down the list in order, and the first match counts |
| Identity check | Against the address lists the operators publish themselves: OpenAI, Anthropic, Perplexity, Amazon, Google and Microsoft; a request using a name but coming from outside that name's list does not count as that crawler |
| When pulled | Log: summed in five passes over ssh at 22:14 to 22:30 on 2026-10-16; the program only read the log, counted in memory and printed counts, and left no files on the server; the verification lists were downloaded at 22:12 to 22:13; the 17 diagnosis requests in section 2 were rechecked read-only at 00:13 to 00:15 on 10-17 against their original User-Agent and the diagnosis record in our product's back end. File checks: our site's llms files, robots.txt, four sitemaps and the product site's home page fetched with curl at 22:18 to 22:28, all after the log window |
| Withheld | Visitors' IP addresses are never given |
| Time zone | The log is recorded in Beijing time (+0800), and every date in this article is in Beijing time |
2. The llms files were requested 210 times: by whom
| Source (by User-Agent) | Requests | Confirmed as ours | Verified as an AI crawler |
|---|
| Declared the name of an AI crawler or search engine spider (Googlebot 22, Claude-User 1) | 23 | 23 | 0 |
Command line and scripts (curl 112, a User-Agent of just Mozilla/5.0 4, a script declaring a website-audit name 14) | 130 | 84 | 0 |
| Three third-party website data collectors (the kind that profile sites' technology or business data; they declare their names and are not AI engines' crawlers) | 14 | 0 | 0 |
| Our product YinJen's website diagnosis (the User-Agent is an ordinary-browser string hard-coded in the diagnosis program; how we recognised it is in reading 2) | 17 | 17 | 0 |
| User-Agent looks like an ordinary browser, cannot be identified | 26 | 0 | 0 |
| Total | 210 | 124 | 0 |
By address: /llms.txt at the root 84, the product site's llms.txt 80, the other two apps' llms.txt 18, the product site's llms-full.txt 14, and /llms-full.txt at the root (no file) 14.
Reading the table:
- Of the 34 names on our list, only two ever fetched an llms file, and neither passed the identity check. The other 32, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider, Meta's several names, Applebot, Baiduspider and Kimi's crawler, are at 0 even when counted by self-declared name: not even requests spoofing these names came for it.
- Of the 43 requests whose User-Agent looks like an ordinary browser, 17 were our own product's website diagnosis, and the other 26 cannot be identified. YinJen's website diagnosis fetches
/llms.txt at the site's root and, if that file is sound, /llms-full.txt as well, using a Windows Chrome User-Agent string hard-coded in the program. These 17 match that string character for character and fall into 9 batches, all between 23:33 on 10 July and 21:04 on 11 July: in every batch the same address fetched the home page with the same string within 20 seconds either side; 8 batches also fetched /.well-known/ai.txt, a path the diagnosis program checks; and 8 of the 9 fetched /llms.txt and then /llms-full.txt within 1 to 3 seconds. The first 8 batches ran in our development environment while these checks were being built: batches 2 to 7 (the 6 between 23:40 and 23:58 on 10 July) already used the two checks that fetch /llms-full.txt and /.well-known/ai.txt, which were only committed to our code repository at 00:06 on 11 July. The last batch came from our own server, and 12 seconds later the server logged a diagnosis of this site under our own account. The 26 we cannot identify all follow one pattern: one request gets a 301 redirect (http to https), then a different address fetches the file, 12 pairs within 1 to 2 seconds and one more pair 4 seconds apart, which looks like a scanner using rotating proxies. They declare no name, so this article does not count them as any crawler. - 46 of the command-line requests carry none of the four clues described in section 4, so this article does not count them as ours. That makes 124 a lower bound.
- There was one more before the window. At 09:12 on 7 July, before the file went live, a verified Claude-User requested
/llms.txt at the root and got a 404. Claude-User is the name Anthropic uses to fetch pages "when a user asks"; whose question triggered this one, the log cannot tell. From the start of the log on 6 June to the cutoff, this is the only time a verified AI crawler requested an llms file.
3. What the same crawlers did fetch
The table counts only verified requests, over the same window:
| Name | Operator | robots.txt (days with requests) | Sitemap (days with requests) | llms files | Successful page fetches |
|---|
| OAI-SearchBot | OpenAI | 681 (100 days) | 0 | 0 | 100 |
| GPTBot | OpenAI | 0 | 348 (100 days) | 0 | 125 |
| ChatGPT-User | OpenAI | 0 | 0 | 0 | 177 |
| ClaudeBot | Anthropic | 503 (52 days) | 2,221 (52 days) | 0 | 129 |
| Claude-User | Anthropic | 12 (7 days) | 0 | 0 | 6 |
| PerplexityBot | Perplexity | 24 (18 days) | 0 | 0 | 49 |
| Amazonbot | Amazon | 2 (2 days) | 0 | 0 | 80 |
| Total for the 7 names above | — | 1,222 | 2,569 | 0 | 666 |
| Reference: Googlebot | Google | 322 (95 days) | 116 (87 days) | 0 | 767 |
| Reference: bingbot | Microsoft | 84 (55 days) | 475 (98 days) | 0 | 630 |
Reading the table:
- robots.txt and llms.txt both sit at the conventional root path; the first is fetched every day, the second not at all. OAI-SearchBot fetched robots.txt on every one of the 100 days, GPTBot fetched sitemaps every day, and ClaudeBot fetched robots.txt on 52 days and sitemaps on 52 days. None of them ever fetched llms.txt, which sits at the root as well.
- The names that fetch pages "when a user asks" did not fetch it either. ChatGPT-User had 177 successful page fetches and Claude-User 6, and neither ever fetched an llms file.
- Not one of the 159 requests under Claude-SearchBot or the 283 under Perplexity-User passed the identity check, so they are not in the table; counted by self-declared name, they also fetched llms files 0 times. GoogleOther (Google's generic crawler, not an AI-only one) is likewise at 0 for llms files among its verified requests.
- The table counts requests, not what was read. robots.txt requests include 301 redirects and 304 "not modified" responses; for example, 446 of OAI-SearchBot's 681 returned 2xx.
4. The 23 requests under declared names: how we know they were ours
Names can be spoofed, so we check identity first; for requests that fail the check, we then look at whether we sent them ourselves. We recognise our own requests by four clues, and an address that meets any one of them counts as ours:
- The User-Agent carries a marker we add when testing (3 addresses).
- Our own work-session records show a fetch of a site file with Claude Code within 2 minutes either side (2 addresses).
- Requests claiming to be Googlebot that fetched llms files from outside Google's list: their times match the records of 5 of our site checks, and each of those checks notes that it also fetched llms files (2 addresses).
- Our own addresses on 9 and 10 October, identified in our previous article from the same log (2 addresses).
With overlaps removed that is 8 addresses, and they sent 107 llms requests; adding the 17 from our product's website diagnosis, recognised in section 2 from the requests themselves, makes 124, more than half of the 210. As for the 23 under declared names:
- Googlebot, 22: spread over 5 stretches between 23 September and 13 October; each time, several llms files were fetched once each within a minute, from 2 addresses outside Google's list. These were left by our site checks, which scan our own site with curl using Googlebot's User-Agent.
- Claude-User, 1: on 6 September, with a User-Agent saying it came from claude-code and an address outside Anthropic's published list; our session records show a fetch of this file with Claude Code a few seconds earlier, and 9 seconds later the same address fetched the file twice more with a User-Agent of just
Mozilla/5.0. That was us checking the file.
So if you want to count who reads the llms.txt on your site, remove yourself first: more than half of our 210 requests were our own.
5. Could a CDN in front of the site be hiding requests?
The origin log only records requests that reach the origin. If a CDN in front of the site had cached llms.txt, crawlers would get the cached copy and the origin would never see those requests. Our situation comes in two parts:
- From 8 July to 8 August (32 days), the site sat behind a Cloudflare proxy. In that period 91.4% of the requests in the log came through Cloudflare's nodes, and the llms files were requested 32 times. When the same address fetched llms.txt several times within seconds or minutes, every request was recorded in the origin log (for example 7 within 25 minutes on 10 July, none carrying query parameters), so there is no sign of the edge cache intercepting them. But this only shows that those requests were not intercepted; it cannot rule out that others were served from cache.
- From 9 August the site no longer went through the proxy, and only 2.3% of requests still came through Cloudflare's nodes after that (we did not work out what kind of clients they were). Looking at those 68 days alone, the verified requests of the 7 names in section 3 fetched robots.txt 964 times, sitemaps 1,825 times and llms files 0 times.
6. What it means for GEO
Ways to use it:
- Publishing an llms.txt is fine, but do not count on it to get AI crawlers to read you. It is cheap, and coding assistants that fetch files on a user's instruction will read it (we fetched ours with Claude Code); but on our site, verifiable AI crawlers did not fetch it once in 100 days. Our own product, YinJen, also checks for llms.txt in its website diagnosis, and lists a missing llms.txt or a missing llms-full.txt as a low-impact item. Key facts such as your brand name, operating entity and scope of capabilities belong in the HTML the server returns directly: what they actually fetch is pages, with 666 verified successful page fetches in the table in section 3. For how to write pages that do not depend on browser-side scripts, see our previous article.
- Get robots.txt and your sitemaps right, and keep them current. These are the machine-readable files AI crawlers actually fetch: OAI-SearchBot fetches robots.txt every day, and GPTBot fetches sitemaps every day. Do not block AI answer engines' crawlers by mistake in robots.txt, and keep your sitemaps complete and updated as you add pages (see the article on robots.txt and sitemaps under Further reading).
- To find out whether anyone reads the llms.txt on your site, count your own log. Pick out requests for llms files by path, group them by User-Agent, then verify them against the address lists the operators publish; remove your own requests first.
Ways not to use it:
- This article makes no promise about indexing, rankings or any other outcome.
Publishing an llms.txt or not does not decide whether you will be cited by AI.
- It does not generalise to other sites. This is one site, one origin log and 100 days; other sites and other periods may differ.
- "0 in the log" does not mean AI cannot use what is in your llms.txt. They may get the same content elsewhere, and they may come for it later; this log only shows that over these 100 days, verifiable AI crawlers did not fetch it from the origin.
7. Caveats
- Nothing links to our llms.txt. None of robots.txt, the product site's home page or the four sitemaps links to it, so crawlers can only find it at the conventional path on their own. Adding a link from pages or robots.txt might change the result; we have not tested that. The llms.txt itself does link to llms-full.txt.
- llms-full.txt has existed for only 52 days. The product site's llms-full.txt first returned 200 on 25 August; its content is the full Chinese documentation of our open API.
- Names are self-declared, and the checks are a snapshot. The checks use the lists as of the day the data was pulled. Bytespider, Baiduspider and Meta's several names were not checked against lists, but counted by self-declared name they also fetched llms files 0 times, so the conclusion is unaffected.
- For the 32 proxied days, caching can only be partly ruled out. See section 5.
- Our own requests were not removed from the totals. The 210 requests in section 2 include our own; the four clues identify 107 of them, and with the 17 from our product's website diagnosis that makes 124, a lower bound.
- China's AI engines barely show up. Kimi's crawler and the Doubao app's in-app browser on our name list never fetched an llms file; crawlers of China's engines are rare in this log to begin with (section 8 of our previous article), so this article cannot say whether they read llms.txt.
- The numbers will date. The log grows every day and the operators' lists change; the numbers here stand only for the moment they were pulled on 16 and 17 October.
Where the data comes from: The log figures are taken from the nginx access log of zhimahang.com for the window 2026-07-08 00:40:51 to 2026-10-16 00:00 Beijing time (104,645 lines), while the one request before the window in reading 4 of section 2 comes from the part of the same log between its start on 6 June and the start of the window; all were summed read-only in five passes between 22:14 and 22:30 on 16 October and rechecked read-only once at 00:13 on 17 October; names are as self-declared in the User-Agent, and identities were checked against the address lists published by OpenAI, Anthropic, Perplexity, Amazon, Google and Microsoft (downloaded at 22:12 to 22:13 on 16 October); "the 32 days behind the Cloudflare proxy" is judged by whether the peer address of each request in the log falls inside the address ranges Cloudflare publishes. The file checks (the sizes of the five llms file addresses, the root and product site copies being byte-for-byte identical, and the absence of any link to llms.txt in robots.txt, the four sitemaps and the product site's home page) come from fetching our own site with curl at 22:18 to 22:28 Beijing time on 16 October, outside the log window; "the root is passed by the gateway to the product site's copy" comes from our code comments and deployment notes. The four clues used to recognise our own requests come from our own site-check records and work-session records; clue 1 in section 4 looks at requests in the same log that carry our test markers, including lines from 16 October itself (after the window). The 17 requests from our product's website diagnosis in section 2 were recognised by the User-Agent hard-coded in the diagnosis program's code, the order in which it fetches files and the other requests in the same batch; the last batch came from the same address our domain resolves to today (our server), and was also matched against the diagnosis record in our product's back end (queried read-only at 00:14 to 00:15 on 17 October). Measurements on a single site, not an industry average.
This article was written by Suzhou ZhiMaHang Technology Co., Ltd. (the YinJen team). The access-log details used in this article have not been published (the log contains visitors' addresses); our public citation datasets are under the GitHub organization ZhiMaHang (verified domain ownership: zhimahang.com). Website: zhimahang.com
Data in this article comes from a read-only count and check of the author's company's own website (log side: the nginx access log from 2026-06-06 to 10-16, Beijing time, with the main window running from 07-08 to 10-15 and the lines from 10-16 used only to recognise our own addresses by our test markers; file side: curl requests to our own site's files on 10-16; also checked against the address lists the operators publish, our own code repositories, the diagnosis records in our own product's back end, and our own check records and work-session records). The text was drafted with AI assistance and reviewed by the author before publication.
Further reading
