Almost everyone doing GEO starts with the same question: can AI engines actually get to my content?
That question usually gets translated into a series of configuration steps—letting AI crawlers through, getting robots right, submitting a sitemap. We have written a dedicated configuration guide for that layer (see Further reading at the end), and this article does not repeat it. It answers only the other half of the question, the part that comes after: once everything is configured, how do you verify it? And what does the result look like?
We ran this on our own site, and the conclusion is rather counterintuitive:
An all-green sitemap does not mean the site has no problems—every problem we found was outside the sitemap.
1. First, Separate Two Surfaces: The Index Surface and the Citation Surface
Many people treat “being indexed by search engines” and “being cited by AI engines” as two ways of saying the same thing. The two are related, but the sets of URLs they look at differ in size:
- The surface you declare: the few dozen entries in your sitemap. This is you telling the engine, “these are what I want you to look at.”
- The surface the engine remembers (the index surface): far larger than the one above. Addresses that were once linked to, once misspelled, or once generated by an old template may all still be remembered by the engine, which periodically comes back to retry them.
- The citation surface: when an AI engine gives a link pointing to you in an answer, it uses the address from its own remembered list, not from your sitemap.
These three surfaces do not coincide, and that is the source of every headache in this article. A historical address that now returns 404 does not exist at all on “the surface you declare”—scan your sitemap ten thousand times and you will never find it. But on the surface the engine remembers, it is alive and well, and once it appears in the citation slot of some AI answer, the user who clicks through gets a 404.
So a self-check that only scans the sitemap is not enough; you have to bring in the failure surface from the engine side and reconcile the two.
2. The Scope of This Measurement
The scope is fixed here, and every number below should be read against it:
| Item | Value |
|---|
| Performed by | Suzhou ZhiMaHang Technology Co., Ltd. (a self-check of our own sites; we state our interest up front) |
| Date | 2026-09-20 (PDT) |
| Target | Our own domain zhimahang.com and its subpath sites |
| Site-side method | Took all four sitemaps declared in robots.txt and requested every URL one by one with a Googlebot UA, scanning for noindex at the same time |
| Site-side sample size | 47 URLs across the four sitemaps, full coverage, no sampling |
| Tool | Local curl (run on a Mac), not a third-party SaaS reading |
| Engine-side method | Readings from the Google Search Console dashboard (property sc-domain:zhimahang.com) |
| Engine-side range | 2026-06-27 ~ 2026-09-18; ⚠️ the report lags by about 2 days, ⛔ so it cannot be used to judge actions taken on 09-19/09-20 |
⚠️ The two sides are not snapshots of the same moment: the site side consists of live requests made on the evening of 09-20, while the engine side is a report as of 09-18. Reconciliation must keep this time gap in mind; otherwise it is easy to misread “an old snapshot the engine remembers” as “the site is broken right now.”
3. Site Side: All 47 Return 200, Zero noindex
Broken down sitemap by sitemap:
| Sitemap | URLs | Non-200 | With noindex |
|---|
/yinjen/sitemap.xml | 33 | 0 | 0 |
/sitemap.xml | 2 | 0 | 0 |
/stride/sitemap.xml | 5 | 0 | 0 |
/sail/sitemap.xml | 7 | 0 | 0 |
| Total | 47 | 0 | 0 |
To restate the scope: measured locally on 2026-09-20, Googlebot UA, all 47 URLs across the four sitemaps, 0 non-200, 0 with noindex.
Widening the scope is a step you cannot skip: the previous round (2026-09-13) scanned only the 33 URLs in /yinjen/sitemap.xml, because that was “the content one”; only this round scanned all four. However many sitemaps robots declares is how many you have to scan; scanning just one amounts to assuming the other three are fine.
At this point, the site side is entirely green. That is when the trouble begins.
4. Engine Side: 68 Indexed, 25 Not Indexed—Broken Down by Reason
In the same GSC report (data range 2026-06-27 ~ 2026-09-18):
- Indexed: 68
- Not indexed: 25
“25 not indexed” means nothing on its own; it has to be broken down by reason:
| Reason | Count | Our reading |
|---|
| Not found (404) | 6 | 🔴 Triaged one by one; see Section 5 |
| Page with redirect | 9 | URLs not pulled individually (not done this round; recorded as is) |
| Excluded by ‘noindex’ tag | 1 | Already handled; see Section 6 |
| Alternate page with proper canonical tag | 1 | 🟢 Expected behavior, not a fault; see Section 7 |
| Crawled - currently not indexed | 7 | On Google’s side; URLs not pulled individually |
| Blocked due to unauthorized request (401) | 1 | 🟢 An API subdomain returning 401 is correct behavior; no fix needed |
| Duplicate without user-selected canonical | 0 | — |
| Discovered - currently not indexed | 0 | 🟢 No crawl backlog |
6+9+1+1+7+1 = 25, consistent with the headline figure.
“Not indexed” is not in itself a health metric. At least two cells in the table above (the 1 URL consolidated via canonical tag and the 1 returning 401) are exactly what we want them to look like; and 0 under “Discovered - currently not indexed” is actually good news. ⇒ Treating 25 as “25 bugs to fix” is the most common misreading: it is a classification table, not a to-do list.
5. The 6 404s Fall into Three Categories of Entirely Different Nature
Correction (2026-09-23): The first version of this section classified Categories A and B as “some step on the site is currently generating URLs that do not exist,” based on their recent last-crawled date (09-18). After publication we checked each one against our server access logs and the site’s code history, and neither the criterion nor the conclusion holds: the -en address in Category B was genuinely live on 2026-06-28 and was then taken down, so it is a historical leftover; for the two addresses in Category A we found no source anywhere on the current site. Google keeps retrying addresses that were taken down long ago, so a recent crawl date does not show that the site still produces them. This section and Sections 9, 10, and 11 have been rewritten accordingly.
This is the most informative cell of the whole self-check, listed item by item (“Last crawled” is the date recorded by GSC):
| # | URL | Last crawled | Category |
|---|
| 1 | zhimahang.com/$ | 2026-09-18 | 🟡 Category A: contains a literal `$`; no current source on the site |
| 2 | yinjen.zhimahang.com/$ | 2026-07-30 | 🟡 Category A: old subdomain + the same `$` |
| 3 | zhimahang.com/yinjen/articles/geo-conversion-tracking-en | 2026-09-18 | 🟡 Category B: an address that was briefly live and then taken down |
| 4 | zhimahang.com/developers | 2026-09-03 | 🟡 Category C: a historical address missing the `/yinjen` prefix |
| 5 | zhimahang.com/articles/public-geo-experiment-issue-2 | 2026-09-01 | 🟡 Category C: same as above |
| 6 | zhimahang.com/about | 2026-07-22 | 🟡 Category C: same as above |
The three categories must be assessed separately—they have different causes and call for different checks.
Category A: URLs containing a literal $ (2)
/$ is not a normal path. The first guess is a template variable that was output without being evaluated—a placeholder like ${...} with only the $ leaking out.
But that is only a hypothesis, and it has to be checked against evidence from the site. We searched every link on the current site’s pages, all four sitemaps, the structured data (JSON-LD), and llms.txt, and none of them produces `/$`; the site’s code and commit history contain no such literal link either. In the server access logs we keep (main domain, from 2026-06-06 onward), the first request for zhimahang.com/$ came on 2026-07-07 (PDT) from Google’s GoogleOther crawler, with no referring page (Referer).
⇒ Reading: no current source on the site; where it was first discovered remains unidentified. Returning 404 for an address that does not exist is the correct response; ⛔ we are not adding redirects just to make the report look clean.
Category B: An address that was briefly live and then taken down (1)
geo-conversion-tracking-en looks like the real slug geo-conversion-tracking (the article on instrumenting the GEO conversion path) with an extra -en tacked on the end—and that is how the first version of this article read it. The access logs overturned that reading: this address was genuinely live on 2026-06-28 (PDT) and returned 200, and Google’s crawlers visited it at the time. Back then our publishing flow posted the English version as a separate article with an -en suffix; the flow was then changed to “one article + ?lang=en,” and this one was taken down—it was live for at least about 7 minutes and at most about 1 hour 41 minutes, and the exact takedown time was not recorded. It has returned 404 ever since, and Google periodically comes back to retry it; the 09-18 crawl recorded by GSC is one of those retries.
⇒ Reading: a historical leftover, not a site defect.
So neither Category A nor Category B is “some step on the site currently generating URLs that do not exist”—that conclusion in the first version of this article does not hold. Like Category C, both belong to kind ② described below.
Category C: Historical addresses Google remembers (3)
What these three have in common is that they are missing the `/yinjen` prefix—the site structure was later changed to a subpath layout, and these are the old addresses from before that change.
⇒ Reading: a historical URL that Google remembers ≠ a current site fault. The GSC 404 list mixes two kinds of things: ① links you are still generating that point to nonexistent pages—these are defects and need fixing; ② old addresses the engine learned long ago and still periodically comes back to retry—normal, and not a fault.
The criterion that tells them apart is “whether anything on the current site still produces the address”—not “whether it returns 404,” and not “whether the last-crawled date is recent.” Category B is the counterexample: taken down almost three months earlier, it was still crawled on 09-18. Both kinds return 404, but only the first calls for action. The check is a full-text search of the current site’s page links, sitemaps, and structured data, plus—where needed—the access logs, to see when the address first appeared and whether it had a referring page.
We ran into kind ① ourselves: in the first version of this article, the three internal links under “Further reading” had an extra /yinjen in their paths, so the rendered paths contained yinjen twice in a row—a source of fresh 404s from the moment it was published. A site search using the criterion above found the source directly. It has been fixed.
6. The noindex URL, and a Page That Is Not in Any Sitemap
The 1 URL excluded by “noindex” is zhimahang.com/geo-check/; GSC records its last crawl as 2026-09-02 and its first detection as 2026-09-04. We removed the noindex from that page on 09-15, and in this round our Googlebot UA test returned 200 with 0 noindex hits—what GSC reports is the old 09-02 snapshot, which is not the same thing as the site’s current state. This is a concrete instance of the “time gap between the two sides” from Section 2.
Following this one turned up a gap nobody had ever logged:
`geo-check/` is not in any of the sitemaps.
robots.txt declares four (/sitemap.xml, /yinjen/sitemap.xml, /stride/sitemap.xml, /sail/sitemap.xml), and this page is in none of them; the “Discovery · Sitemaps” field in GSC’s URL Inspection shows exactly None / Temporary processing error.
⇒ This is precisely the concrete form of “the three surfaces do not coincide” from Section 1: a page we have already decided to open to crawling does not exist on “the surface we actively declare,” so it can only get into the queue through manual submission, rather than being reliably rediscovered via the sitemap. It has been logged as a pending site change; ⛔ likewise, we are not announcing it as done.
7. ?f= Attribution Parameters Being “Consolidated” Is Expected Behavior, Not a Fault
The “Alternate page with proper canonical tag” cell appeared for the first time this round, with just 1 URL:
| URL | Last crawled | First detected |
|---|
zhimahang.com/?f=yt6 | 2026-09-08 | 2026-09-14 |
?f=yt6 is one of our attribution-parameter links (used to distinguish the sources of return traffic from off-site placements). After crawling it, Google consolidated it into the homepage according to the canonical tag, so it is not indexed separately—which is exactly the result we want. Attribution parameters exist to mark traffic sources, not to make the engine store N copies of the homepage. ⛔ No fix is needed, and there is no need to click “Validate Fix” either—there is nothing to fix.
🔵 One positive side note: it shows that attribution links carrying `?f=` really are being picked up off-site, so the attribution layer is alive.
⇒ This cell is a reminder: GSC puts all sorts of “not indexed” under the same heading, and some of them are things you designed yourself. When you see a new cell, the first step is to classify it, not to rush to fix it.
8. Two Pitfalls That Will Lead You to False Conclusions
We actually stepped in both of these (Pitfall One during this round’s 09-20 scan, Pitfall Two during the previous round’s 09-13 scan). Without them written down here, none of the numbers above would be reproducible.
Pitfall One: When a single curl fails, retry before declaring it dead
During this scan, zhimahang.com/sail/cases returned `000` on the first pass (could not connect / handshake failed), and returned 200 with noindex 0 on retry. This week, similar transient TLS failures from this machine were recorded 3 times in total: 2 on the evening of 09-20 (this 1 in the scan, plus an online recheck the same evening) and 1 on 09-17 (a push-API call).
⇒ Verdict: a transient TLS failure between this machine and the site is not a site fault.
⇒ This led to a rule: a single failure must be retried before it is declared dead. Had we logged the first-pass results, the conclusion would have become “1 of the 47 is unreachable”—a false fault manufactured purely by local network jitter, which would also send people chasing a problem that does not exist.
⚠️ The scope of this point extends only to this batch of scans: what it shows is that “our local network had known jitter on 09-20”; ⛔ it does not constitute any general conclusion about the site’s availability.
Pitfall Two: BSD sed does not support BRE \?, and getting it wrong silently returns 000 for everything
To strip the <loc> tags from a sitemap and extract the URLs, the correct form is:
sed -E 's#</?loc>##g'
Note the `-E` (extended regular expressions). If you write it as BRE with \? on the BSD sed that ships with macOS, it will not throw an error—it treats \? as a literal, the tags are not fully stripped, the “URLs” you request still carry angle-bracket fragments, and curl returns `000` for all of them.
⇒ This is the most dangerous pitfall in this article, because the failure mode is silent: you get a table saying “all 47 unreachable,” which looks like a major incident but is actually one wrong line of sed. ⇒ A general criterion: when a batch of results “fails uniformly,” suspect your own pipeline first, and the thing being tested second.
9. How to Copy This Self-Check (Four Steps)
It does not depend on any particular tool and can be run entirely from the command line:
- Pull every sitemap declaration from robots.txt—take however many there are; ⛔ don’t take only “the content one.”
- Request each URL with a crawler UA and record the status code; a failure on any URL must be retried before it is declared dead.
- Scan for `noindex` in the same pass—a page that returns 200 but carries
noindex does not exist as far as the index surface is concerned, and looking only at status codes will miss it. - Bring in the engine-side failure surface and reconcile—break “Not indexed” down by reason, and for each 404 check whether anything on the current site still produces it: if so, it is a defect to fix; if not, it is an old address the engine remembers.
Step 4 is the easiest to skip and also the most valuable. The first three steps only prove that “what I declared is still alive”; only Step 4 can tell you what exists beyond what you declared. The reconciliation result, in one sentence:
Not one of GSC’s 6 404s and 1 noindex falls among the current 47 sitemap URLs.
10. Boundaries: What Was Observed, How We Read It, and What Does Not Hold
Observed: In local measurements on 2026-09-20, all 47 URLs across the four sitemaps showed 0 non-200 and 0 with noindex under a Googlebot UA; for the same period, GSC (range 2026-06-27~2026-09-18) reported 68 indexed and 25 not indexed, including 6 404s, with zero overlap with those 47.
Reading: The “surface we actively declare” is clean; all 6 of GSC’s 404s lie outside what is declared, and after checking each one, none of them is being produced by the current site—Category B is an address that was briefly live on 06-28, Category C consists of old addresses from before a redesign, and for Category A no source can be found on the current site. (The first version read this as “pointing to a step that is currently producing bad links”; that has been corrected—see Section 5.)
⛔ None of the following holds in this article:
- ⛔ This does not imply causation. This article performed no causal test between the all-green site side and the index counts; they are merely two readings from the same day.
- ⛔ No indexing, citation, or ranking outcome is promised. Cleaning up the sitemap and clearing out 404s is reducing known failures, not obtaining a particular result; no party can guarantee how search engines or AI engines index or cite content.
- ⛔ A single snapshot does not constitute a trend; no single-window reading can serve as a baseline.
- ⛔ This cannot be extrapolated to other sites. A scale of 47 URLs and a structure of four sitemaps are our own shape; on a different site the conclusions will most likely not hold, but the method can be copied.
- What was not done, recorded as is: the 9 “Page with redirect” URLs and the 7 “Crawled - currently not indexed” URLs were not pulled individually this round and remain open.
- ⛔ A recent “last crawled” date does not mean the site still produces the address. The first version of this article misread exactly this: Google periodically retries addresses that were taken down long ago.
11. Back to GEO: Why This Is Worth Doing Once a Week
From a pure SEO-operations standpoint, this article is an inspection record. In a GEO context, it answers a question that comes earlier: when an AI engine is about to cite you in an answer, where does the URL in its hands come from?
Not from your sitemap, but from the address table the engine has built up on its own—which mixes your current pages, old addresses from before a redesign, addresses that were only briefly live before being taken down, and addresses whose source can no longer be traced.
So “being cited” has to clear at least two hurdles: the content has to be selected (the focus of most GEO discussion), and the selected address has to actually open (something almost nobody checks). When the second hurdle fails, the symptom is not “not mentioned” but “mentioned, but clicking through gives a 404”—completely invisible in mention-rate readings, because the mention really did happen.
This is also why we have made it a fixed once-a-week routine: what really needs fixing is bad links the site itself is currently producing, and they can appear with any new publication—the first version of this article carried three of them (see Section 5). Checking each newly appearing 404 against its source on the site every week is how you catch and fix them early.
Sources and scope: Site-side figures come from measurements of our own production sites made with curl on a local Mac on 2026-09-20 (PDT) (Googlebot UA, all 47 URLs across four sitemaps, no sampling); engine-side figures come from Google Search Console dashboard readings for the property sc-domain:zhimahang.com, range 2026-06-27 ~ 2026-09-18, with a reporting lag of about 2 days. The checks on Categories A and B in Section 5 additionally used our server access logs (main domain, retained from 2026-06-06) and the site’s code commit history, checked on 2026-09-23 (PDT). See Section 10 for boundaries.
YinJen (zhimahang.com/yinjen) by Suzhou ZhiMaHang Technology Co., Ltd.: a desktop application covering 12 AI engines, with Visibility Score ratings and paragraph-level optimization recommendations. Creator plan ¥29.9/month, with a 14-day free trial.
Data in this article comes from a self-check of the author's company's own website (site side: local curl requests on 2026-09-20 PDT; search-engine side: Google Search Console readings). The text was drafted with AI assistance and reviewed by the author before publication.
Further reading
