TL;DR: Before an AI can cite you, it has to read you. Let the AI answer-engine crawlers through at Cloudflare, declare the right Content-Signal in robots.txt (search / ai-input / ai-train all set to yes, zero Disallow on real content — only /login, /api, /admin are blocked), then submit your sitemap to Google, Bing and Baidu. No indexing, no citation.

Why "being indexed" is the GEO starting line

The goal of GEO (Generative Engine Optimization) is for your content to show up in — and be cited by — the answers from ChatGPT, Claude, Gemini, Perplexity, Doubao and other AI engines. But people skip a prerequisite: an AI engine has to be able to *read* your site before it can ever *cite* you. If you lock AI crawlers out, or search engines never indexed your pages, even great content stays invisible to AI.

So the first mile of GEO is really two unglamorous things: let the right crawlers in, and get the search/answer engines to index you. Below is the full sequence we ran on a real site — five steps you can copy.

Step 1 — Let the AI answer-engine crawlers in (Cloudflare)

Many sites (especially behind Cloudflare) block AI crawlers wholesale by default. Cloudflare's managed "block AI training bots" switch, once on, locks out ClaudeBot, GPTBot, Google-Extended, Amazonbot and more — a set that mixes training crawlers with the ones that fetch content for AI answers in real time.

What we did:

Flipped that master switch from "block on all pages" to "don't block," which makes Cloudflare's per-crawler toggles controllable again;
Confirmed Bot Fight Mode and AI Labyrinth weren't challenging these verified bots either.

In one line: open the network-layer door first — otherwise nothing you write in robots.txt matters.

Step 2 — Send the right content signals via robots.txt

Once the network layer is open, the thing that actually decides "which crawler gets in and what it may do" is robots.txt. Two points.

1) Stop `Disallow: /`-ing the answer engines. ClaudeBot, GPTBot and Google-Extended all *obey* robots.txt — so a blanket Disallow keeps them out even when the WAF is wide open. Remove those blanket blocks.

2) State your intent positively with Content Signals. This is an emerging standard (contentsignals.org): one line declaring what crawlers may do with your content:

search=yes: allow building a search index;
ai-input=yes: allow your content as input to AI answers (RAG, grounding, AI overviews) — exactly what GEO wants;
ai-train=yes/no: whether to allow model training (your call).

A good root robots.txt looks roughly like this:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /login
Disallow: /api/
Disallow: /admin/

Sitemap: https://your-domain/sitemap.xml

Three things to note: ① only utility paths like login/api/admin are blocked, the real content is fully open; ② the GEO bonus signal ai-input=yes is explicit; ③ the Sitemap: line lets every crawler — including Baidu, which (as we'll see) can't always be submitted to by hand — discover your sitemap on its own.

Step 3 — Get indexed in Google Search Console

Add a Domain property (the whole domain) in GSC and verify it with a DNS TXT record. One Domain property covers the root, www, subdomains and subpaths at once — simpler than URL-prefix properties.
Submit your sitemap — "Success · N pages discovered" is what you want.
For your key pages (home, features, pricing, download), use URL Inspection → Request Indexing to push them into the priority crawl queue.

One trap: if you're moving from an old subdomain to a new subpath, don't use GSC's "Change of Address" tool — it only supports subdomain → subdomain. Rely on 301 redirects and keep the old property alive.

Step 4 — Bing

Bing Webmaster Tools lets you sign in with a Google account (no Microsoft signup needed). Add the site and verify via CNAME (a DNS record pointing to verify.bing.com — remember to set it to "DNS only, not proxied"), then submit the same sitemap. Bing also has IndexNow for instant submission, which is worth turning on.

Step 5 — Baidu, and the ICP-filing gate

Baidu matters for Chinese-language sites, but it plays by its own rules:

After adding the site, verify it (file / HTML tag / CNAME);
Sitemap submission requires an ICP filing — without one, the sitemap quota is 0 and even the push API returns over quota;
Before you have a filing, use "manual submission" to push a few core URLs a day so Baiduspider starts crawling;
That Sitemap: line in robots.txt also helps Baiduspider find your sitemap on its own.

So for Chinese-site GEO/SEO, file your ICP as early as possible — it's the master valve on the Baidu side.

Takeaways

"Allowing" is layered: opening the network layer (WAF) is not the same as opening robots.txt. Open both — good crawlers obey robots.txt, and a bad config locks them out.
robots.txt is your GEO statement of intent: it isn't only about "what's forbidden," it's where you use Content Signals to say "you're welcome to use my content in answers."
One `Sitemap:` line is worth a thousand words: it lets every engine discover your content map on its own, especially the engines you can't submit to by hand.
Being indexed isn't being cited — but it's the prerequisite. All of this just opens the door. What actually decides whether you make it into an AI's answer is how structured and citable your content is — and that's the second half of GEO.

Opening the door is only the start. That second half — getting your content actually understood, selected and cited by AI answer engines — is exactly what YinJen works on.

Further reading: getting read and indexed is just the kickoff — once you're cited, how do you measure the real conversions it drives? See GEO Needs Analytics Too: Measuring the Full Loop from AI Citation to Real Conversion, or see how YinJen handles the whole loop.

GEO Step Zero: Getting Your Site Read and Indexed by AI Answer Engines