Almost everyone doing GEO starts with the same question: can AI engines actually get to my content?
That question usually gets translated into a series of configuration steps—letting AI crawlers through, getting robots right, submitting a sitemap. We have written a dedicated configuration guide for that layer (see Further reading at the end), and this article does not repeat it. It answers only the other half of the question, the part that comes after: once everything is configured, how do you verify it? And what does the result look like?
We ran this on our own site, and the conclusion is rather counterintuitive:
An all-green sitemap does not mean the site has no problems—every problem we found was outside the sitemap.
1. First, Separate Two Surfaces: The Index Surface and the Citation Surface
Many people treat “being indexed by search engines” and “being cited by AI engines” as two ways of saying the same thing. The two are related, but the sets of URLs they look at differ in size:
- The surface you declare: the few dozen entries in your sitemap. This is you telling the engine, “these are what I want you to look at.”
- The surface the engine remembers (the index surface): far larger than the one above. Addresses that were once linked to, once misspelled, or once generated by an old template may all still be remembered by the engine, which periodically comes back to retry them.
- The citation surface: when an AI engine gives a link pointing to you in an answer, it uses the address from its own remembered list, not from your sitemap.
These three surfaces do not coincide, and that is the source of every headache in this article. A historical address that now returns 404 does not exist at all on “the surface you declare”—scan your sitemap ten thousand times and you will never find it. But on the surface the engine remembers, it is alive and well, and once it appears in the citation slot of some AI answer, the user who clicks through gets a 404.
So a self-check that only scans the sitemap is not enough; you have to bring in the failure surface from the engine side and reconcile the two.
2. The Scope of This Measurement
The scope is fixed here, and every number below should be read against it:
| Item | Value |
|---|
| Performed by | Suzhou ZhiMaHang Technology Co., Ltd. (a self-check of our own sites; we state our interest up front) |
| Date | 2026-09-20 (PDT) |
| Target | Our own domain zhimahang.com and its subpath sites |
| Site-side method | Took all four sitemaps declared in robots.txt and requested every URL one by one with a Googlebot UA, scanning for noindex at the same time |
| Site-side sample size | 47 URLs across the four sitemaps, full coverage, no sampling |
| Tool | Local curl (run on a Mac), not a third-party SaaS reading |
| Engine-side method | Readings from the Google Search Console dashboard (property sc-domain:zhimahang.com) |
| Engine-side range | 2026-06-27 ~ 2026-09-18; ⚠️ the report lags by about 2 days, ⛔ so it cannot be used to judge actions taken on 09-19/09-20 |
⚠️ The two sides are not snapshots of the same moment: the site side consists of live requests made on the evening of 09-20, while the engine side is a report as of 09-18. Reconciliation must keep this time gap in mind; otherwise it is easy to misread “an old snapshot the engine remembers” as “the site is broken right now.”
3. Site Side: All 47 Return 200, Zero noindex
Broken down sitemap by sitemap:
| Sitemap | URLs | Non-200 | With noindex |
|---|
/yinjen/sitemap.xml | 33 | 0 | 0 |
/sitemap.xml | 2 | 0 | 0 |
/stride/sitemap.xml | 5 | 0 | 0 |
/sail/sitemap.xml | 7 | 0 | 0 |
| Total | 47 | 0 | 0 |
To restate the scope: measured locally on 2026-09-20, Googlebot UA, all 47 URLs across the four sitemaps, 0 non-200, 0 with noindex.
Widening the scope is a step you cannot skip: the previous round (2026-09-13) scanned only the 33 URLs in /yinjen/sitemap.xml, because that was “the content one”; only this round scanned all four. However many sitemaps robots declares is how many you have to scan; scanning just one amounts to assuming the other three are fine.
At this point, the site side is entirely green. That is when the trouble begins.
4. Engine Side: 68 Indexed, 25 Not Indexed—Broken Down by Reason
In the same GSC report (data range 2026-06-27 ~ 2026-09-18):
- Indexed: 68
- Not indexed: 25
“25 not indexed” means nothing on its own; it has to be broken down by reason:
| Reason | Count | Our reading |
|---|
| Not found (404) | 6 | 🔴 Triaged one by one; see Section 5 |
| Page with redirect | 9 | URLs not pulled individually (not done this round; recorded as is) |
| Excluded by ‘noindex’ tag | 1 | Already handled; see Section 6 |
| Alternate page with proper canonical tag | 1 | 🟢 Expected behavior, not a fault; see Section 7 |
| Crawled - currently not indexed | 7 | On Google’s side; URLs not pulled individually |
| Blocked due to unauthorized request (401) | 1 | 🟢 An API subdomain returning 401 is correct behavior; no fix needed |
| Duplicate without user-selected canonical | 0 | — |
| Discovered - currently not indexed | 0 | 🟢 No crawl backlog |
6+9+1+1+7+1 = 25, consistent with the headline figure.
“Not indexed” is not in itself a health metric. At least two cells in the table above (the 1 URL consolidated via canonical tag and the 1 returning 401) are exactly what we want them to look like; and 0 under “Discovered - currently not indexed” is actually good news. ⇒ Treating 25 as “25 bugs to fix” is the most common misreading: it is a classification table, not a to-do list.
5. The 6 404s Fall into Three Categories of Entirely Different Nature
This is the most informative cell of the whole self-check, listed item by item (“Last crawled” is the date recorded by GSC):
| # | URL | Last crawled | Category |
|---|
| 1 | zhimahang.com/$ | 2026-09-18 | 🔴 Category A: contains a literal `$` |
| 2 | yinjen.zhimahang.com/$ | 2026-07-30 | 🔴 Category A: old subdomain + the same `$` |
| 3 | zhimahang.com/yinjen/articles/geo-conversion-tracking-en | 2026-09-18 | 🔴 Category B: a slug that has never existed on the site |
| 4 | zhimahang.com/developers | 2026-09-03 | 🟡 Category C: a historical address missing the `/yinjen` prefix |
| 5 | zhimahang.com/articles/public-geo-experiment-issue-2 | 2026-09-01 | 🟡 Category C: same as above |
| 6 | zhimahang.com/about | 2026-07-22 | 🟡 Category C: same as above |
The three categories must be assessed separately, and they call for opposite responses.
Category A: URLs containing a literal $ (2)
/$ is not a normal path; a normal link-generation process cannot produce this shape. It looks more like a template variable that was output without being evaluated—a placeholder like ${...} with only the $ leaking out.
The key is the date: zhimahang.com/$ was last crawled on 2026-09-18, very recently. ⇒ Something is currently producing bad links; this is not a historical leftover: the signature of a historical leftover is a crawl date that keeps getting older (like the old subdomain one at 07-30), not one that is still being freshly crawled this month.
Category B: A slug that has never existed on the site (1)
geo-conversion-tracking-en has never existed on our site. English versions use the ?lang=en query parameter; there is no slug form with an `-en` suffix at all. Its last crawl was also 2026-09-18.
⇒ This is not “an old page that was deleted” but an address that was pieced together—the correct slug is geo-conversion-tracking, which really does exist on the site (the article on instrumenting the GEO conversion path), with an extra -en tacked on the end.
Taken together, Categories A and B point to the same conclusion: some step on the site is currently generating URLs that do not exist. This is the only “real site-side defect” lead from this self-check, and it is the entire value of this exercise over simply scanning the sitemap—not one of these three is among those 47.
⇒ Handling: not closed out; logged as “suspected link-generation defect, source to be located.” The lines of investigation are the “Referring page” on that URL’s detail page in GSC + a full-text search of the site. ⛔ We are not announcing it as fixed.
Category C: Historical addresses Google remembers (3)
What these three have in common is that they are missing the `/yinjen` prefix—the site structure was later changed to a subpath layout, and these are the old addresses from before that change. Their crawl dates are 09-03, 09-01, and 07-22 respectively; none of them keeps getting fresher this month.
⇒ Reading: a historical URL that Google remembers ≠ a current site fault. The GSC 404 list mixes two kinds of things: ① links you are still generating that point to nonexistent pages—these are defects and need fixing; ② old addresses the engine learned long ago and still periodically comes back to retry—normal, and not a fault.
The criterion that tells them apart is “whether the last-crawled date is still getting fresher,” not “whether it returns 404.” Both return 404, but only the first kind calls for action.
6. The noindex URL, and a Page That Is Not in Any Sitemap
The 1 URL excluded by “noindex” is zhimahang.com/geo-check/; GSC records its last crawl as 2026-09-02 and its first detection as 2026-09-04. We removed the noindex from that page on 09-15, and in this round our Googlebot UA test returned 200 with 0 noindex hits—what GSC reports is the old 09-02 snapshot, which is not the same thing as the site’s current state. This is a concrete instance of the “time gap between the two sides” from Section 2.
Following this one turned up a gap nobody had ever logged:
`geo-check/` is not in any of the sitemaps.
robots.txt declares four (/sitemap.xml, /yinjen/sitemap.xml, /stride/sitemap.xml, /sail/sitemap.xml), and this page is in none of them; the “Discovery · Sitemaps” field in GSC’s URL Inspection shows exactly None / Temporary processing error.
⇒ This is precisely the concrete form of “the three surfaces do not coincide” from Section 1: a page we have already decided to open to crawling does not exist on “the surface we actively declare,” so it can only get into the queue through manual submission, rather than being reliably rediscovered via the sitemap. It has been logged as a pending site change; ⛔ likewise, we are not announcing it as done.
7. ?f= Attribution Parameters Being “Consolidated” Is Expected Behavior, Not a Fault
The “Alternate page with proper canonical tag” cell appeared for the first time this round, with just 1 URL:
| URL | Last crawled | First detected |
|---|
zhimahang.com/?f=yt6 | 2026-09-08 | 2026-09-14 |
?f=yt6 is one of our attribution-parameter links (used to distinguish the sources of return traffic from off-site placements). After crawling it, Google consolidated it into the homepage according to the canonical tag, so it is not indexed separately—which is exactly the result we want. Attribution parameters exist to mark traffic sources, not to make the engine store N copies of the homepage. ⛔ No fix is needed, and there is no need to click “Validate Fix” either—there is nothing to fix.
🔵 One positive side note: it shows that attribution links carrying `?f=` really are being picked up off-site, so the attribution layer is alive.
⇒ This cell is a reminder: GSC puts all sorts of “not indexed” under the same heading, and some of them are things you designed yourself. When you see a new cell, the first step is to classify it, not to rush to fix it.
8. Two Pitfalls That Will Lead You to False Conclusions
We actually stepped in both of these (Pitfall One during this round’s 09-20 scan, Pitfall Two during the previous round’s 09-13 scan). Without them written down here, none of the numbers above would be reproducible.
Pitfall One: When a single curl fails, retry before declaring it dead
During this scan, zhimahang.com/sail/cases returned `000` on the first pass (could not connect / handshake failed), and returned 200 with noindex 0 on retry. This week, similar transient TLS failures from this machine were recorded 3 times in total: 2 on the evening of 09-20 (this 1 in the scan, plus an online recheck the same evening) and 1 on 09-17 (a push-API call).
⇒ Verdict: a transient TLS failure between this machine and the site is not a site fault.
⇒ This led to a rule: a single failure must be retried before it is declared dead. Had we logged the first-pass results, the conclusion would have become “1 of the 47 is unreachable”—a false fault manufactured purely by local network jitter, which would also send people chasing a problem that does not exist.
⚠️ The scope of this point extends only to this batch of scans: what it shows is that “our local network had known jitter on 09-20”; ⛔ it does not constitute any general conclusion about the site’s availability.
Pitfall Two: BSD sed does not support BRE \?, and getting it wrong silently returns 000 for everything
To strip the <loc> tags from a sitemap and extract the URLs, the correct form is:
sed -E 's#</?loc>##g'
Note the `-E` (extended regular expressions). If you write it as BRE with \? on the BSD sed that ships with macOS, it will not throw an error—it treats \? as a literal, the tags are not fully stripped, the “URLs” you request still carry angle-bracket fragments, and curl returns `000` for all of them.
⇒ This is the most dangerous pitfall in this article, because the failure mode is silent: you get a table saying “all 47 unreachable,” which looks like a major incident but is actually one wrong line of sed. ⇒ A general criterion: when a batch of results “fails uniformly,” suspect your own pipeline first, and the thing being tested second.
9. How to Copy This Self-Check (Four Steps)
It does not depend on any particular tool and can be run entirely from the command line:
- Pull every sitemap declaration from robots.txt—take however many there are; ⛔ don’t take only “the content one.”
- Request each URL with a crawler UA and record the status code; a failure on any URL must be retried before it is declared dead.
- Scan for `noindex` in the same pass—a page that returns 200 but carries
noindex does not exist as far as the index surface is concerned, and looking only at status codes will miss it. - Bring in the engine-side failure surface and reconcile—break “Not indexed” down by reason, and for each 404 decide whether it is “a defect still getting fresher” or “an old address the engine remembers.”
Step 4 is the easiest to skip and also the most valuable. The first three steps only prove that “what I declared is still alive”; only Step 4 can tell you what exists beyond what you declared. The reconciliation result, in one sentence:
Not one of GSC’s 6 404s and 1 noindex falls among the current 47 sitemap URLs.
10. Boundaries: What Was Observed, How We Read It, and What Does Not Hold
Observed: In local measurements on 2026-09-20, all 47 URLs across the four sitemaps showed 0 non-200 and 0 with noindex under a Googlebot UA; for the same period, GSC (range 2026-06-27~2026-09-18) reported 68 indexed and 25 not indexed, including 6 404s, with zero overlap with those 47.
Reading: The “surface we actively declare” is clean; the problems all lie outside what is declared—an address with a literal $ and a nonexistent -en slug each had one URL still being crawled on 09-18, pointing to a step that is currently producing bad links.
⛔ None of the following holds in this article:
- ⛔ This does not imply causation. This article performed no causal test between the all-green site side and the index counts; they are merely two readings from the same day.
- ⛔ No indexing, citation, or ranking outcome is promised. Cleaning up the sitemap and clearing out 404s is reducing known failures, not obtaining a particular result; no party can guarantee how search engines or AI engines index or cite content.
- ⛔ A single snapshot does not constitute a trend; no single-window reading can serve as a baseline.
- ⛔ This cannot be extrapolated to other sites. A scale of 47 URLs and a structure of four sitemaps are our own shape; on a different site the conclusions will most likely not hold, but the method can be copied.
- What was not done, recorded as is: the 9 “Page with redirect” URLs and the 7 “Crawled - currently not indexed” URLs were not pulled individually this round and remain open.
11. Back to GEO: Why This Is Worth Doing Once a Week
From a pure SEO-operations standpoint, this article is an inspection record. In a GEO context, it answers a question that comes earlier: when an AI engine is about to cite you in an answer, where does the URL in its hands come from?
Not from your sitemap, but from the address table the engine has built up on its own—which mixes your current pages, old addresses from before a redesign, and addresses that some template accidentally generated and that never existed.
So “being cited” has to clear at least two hurdles: the content has to be selected (the focus of most GEO discussion), and the selected address has to actually open (something almost nobody checks). When the second hurdle fails, the symptom is not “not mentioned” but “mentioned, but clicking through gives a 404”—completely invisible in mention-rate readings, because the mention really did happen.
This is also why we have made it a fixed once-a-week routine: for the “currently produced defects” in Categories A and B, the criterion is whether the crawl date is still getting fresher, and that criterion only works when there are historical readings—do it just once, and you cannot tell which ones are new.
Sources and scope: Site-side figures come from measurements of our own production sites made with curl on a local Mac on 2026-09-20 (PDT) (Googlebot UA, all 47 URLs across four sitemaps, no sampling); engine-side figures come from Google Search Console dashboard readings for the property sc-domain:zhimahang.com, range 2026-06-27 ~ 2026-09-18, with a reporting lag of about 2 days. See Section 10 for boundaries.
YinJen (zhimahang.com/yinjen) by Suzhou ZhiMaHang Technology Co., Ltd.: a desktop application covering 12 AI engines, with Visibility Score ratings and paragraph-level optimization recommendations. Creator plan ¥29.9/month, with a 14-day free trial.
Data in this article comes from a self-check of the author's company's own website (site side: local curl requests on 2026-09-20 PDT; search-engine side: Google Search Console readings). The text was drafted with AI assistance and reviewed by the author before publication.
Further reading
