7
Here is a scenario that plays out more often than people realize. A company spends months building out exactly the kind of content we have talked about in this series – deep, well-structured, genuinely authoritative articles that answer real questions thoroughly. They do the topic mapping. They build the topic clusters with real depth. They even land some genuine earned media coverage.
And then none of it shows up in AI answers. Not because the content is not good enough. Because the AI systems that would retrieve and cite it cannot actually get to it in the first place.
This is the unglamorous, deeply important part of GEO that nobody enjoys talking about, because it does not feel like strategy. It feels like plumbing. But if the pipes are broken, it does not matter how good the water is. This article is about making sure your pipes work – making sure that when an AI crawler comes looking for your content, it can find it, read it, and understand it without friction.
I also want to deal head-on with some of the noise in this space, because there has been a lot of advice floating around over the past year that ranges from genuinely useful to actively distracting, and I think it is worth being clear about which is which.
Step One: Make Sure the Crawlers Can Actually Get In
This sounds almost too basic to mention, and yet it is the single most common technical issue holding brands back from AI visibility. Many sites – sometimes deliberately, often by accident – are blocking the very crawlers that would bring their content into AI systems.
There are now several AI crawlers you need to be aware of, each associated with a different AI system. OpenAI’s GPTBot, Anthropic’s ClaudeBot, and PerplexityBot are among the most active, with GPTBot alone generating hundreds of millions of monthly requests across the web according to crawler traffic analyses. Google’s AI Overviews largely rely on Google’s existing crawling infrastructure, while other systems use their own dedicated bots.
Your robots
txt file is the first thing to check. It is a simple text file, but it has outsized importance because it is the first thing many of these crawlers check before doing anything else. If your robots.txt disallows a crawler – whether through an old blanket rule that predates AI crawlers, or a more recent rule added out of caution about AI training – that crawler will not retrieve your content, full stop.
The decision about which AI crawlers to allow is not purely technical. There is a legitimate business question buried in here about whether you want your content used to train AI models versus whether you want it retrieved for real-time answers, and some site owners have understandably mixed feelings about the former while still wanting the latter. Some crawlers are starting to distinguish between these purposes, and there is ongoing work at the standards level – through bodies like the IETF – to build clearer distinctions into the robots.txt protocol itself. For now, the practical reality is that if visibility in AI answers matters to your business, blocking the major AI crawlers outright works directly against that goal.
Beyond robots.txt, check whether your CDN or security configuration is silently rejecting AI bot traffic. This happens more often than people expect, particularly with services like Cloudflare, where bot-protection settings configured for security reasons can inadvertently block legitimate AI crawlers along with the malicious traffic they were designed to stop.
Step Two: If It Is Hidden Behind JavaScript, Many AI Systems Cannot See It
This is the issue that surprises people most, because it is invisible if you are looking at your site the normal way – through a browser, where JavaScript renders everything beautifully and you see exactly the page you intended.
Many AI crawlers do not render JavaScript the way a browser does. They request the raw HTML of a page, and if your important content – your article text, your key data, your product information – is loaded dynamically through JavaScript after the initial page load, a crawler that only reads the raw HTML may see an essentially empty page.
The practical test is straightforward: view your page’s source HTML directly, before any JavaScript executes, and see what is actually there. If your core content is present in that raw HTML, you are in reasonable shape. If the raw HTML is mostly empty divs and placeholder elements that get filled in by JavaScript afterward, that is a real problem for AI crawlability – and for that matter, it can be a problem for traditional search crawlability too, though Google has gotten better at handling this over the years than many AI crawlers currently are.
The fix for this is server-side rendering, or one of its variants – static site generation, where pages are pre-built as complete HTML, or incremental static regeneration, which combines pre-building with periodic updates. If a full migration to server-side rendering is not realistic for your situation, dynamic rendering tools that serve a pre-rendered version of your page specifically to bots can bridge the gap, though the cleanest long-term solution is making sure your content exists in the HTML from the start.
Step Three: Schema Markup – What Actually Helps and What Is Overkill
Schema markup, also called structured data, is a way of explicitly labeling the parts of your content so that machines do not have to guess what something is. This article is an Article. This is its author. This is a Frequently Asked Questions section, and here are the specific question-and-answer pairs within it.
There is reasonable evidence that this helps. Some implementation guides report that sites with comprehensive structured data across their key page types see meaningfully more appearances in AI Overview results compared to sites without it. The logic makes sense: when you remove ambiguity about what a piece of content is, you make it easier for any system – search engine or AI – to use that content appropriately.
That said, I want to flag something important here, because the GEO advice ecosystem has occasionally overstated this. Google’s own 2026 guidance on generative AI search optimization has been fairly direct that excessive structured data is not a requirement for appearing in AI Overviews or AI Mode, and that the foundation remains the same things that have always mattered for good SEO – genuinely useful, well-written content that real people would want to read.
My honest read on this is that schema markup is a genuinely useful, relatively low-cost technical investment – particularly Article schema for your content, FAQPage schema for genuine FAQ sections, and Organization schema that clearly establishes who you are. It is not, however, a substitute for the content quality and structural clarity we discussed in earlier articles, and it is not going to rescue thin or poorly organized content. Think of it as a helpful accelerant on top of a fire that is already burning, not a fire on its own.
The llms.txt Question: Useful Emerging Standard or Overhyped Distraction?
If you have spent any time researching technical GEO over the past year, you have almost certainly encountered llms.txt – a proposed standard, originally put forward by Jeremy Howard in 2024, that works something like a robots.txt file but is designed to give AI systems a clean, curated overview of a site’s most important content and structure, often in markdown format.
The idea has gained real traction, particularly among developer-tool companies. Platforms like Stripe, Vercel, and various documentation providers have been experimenting with llms.txt files, and there is a reasonable case for why it works well in that specific context: when an AI coding assistant is trying to figure out how to help a developer integrate with an API, having a clean, curated map of the documentation is genuinely useful, and the assistant can follow that map directly to the most relevant reference material.
Here is where I want to be careful, though
Google’s own 2026 guidance explicitly states that you do not need to create llms.txt files, AI-specific content rewrites, or special markdown versions of your content to appear in Google’s generative AI search features. That is a fairly direct statement from the company that, for many sites, controls the AI search experience most of their audience will encounter.
My honest take, reconciling these different signals: llms.txt appears to have real, demonstrated value for developer-facing sites and technical documentation, where AI coding assistants are a meaningful part of your audience and a curated map of your docs genuinely helps those assistants do their job. For general content sites – blogs, marketing sites, informational resources – the evidence that llms.txt meaningfully affects whether ChatGPT or Perplexity cites your content is much thinner, and Google has explicitly said it does not factor into their systems.
If you run a developer-focused product with substantial documentation, implementing an llms.txt file is a low-cost experiment that fits a pattern of genuine, demonstrated value. If you run a general content site and you are choosing between spending a day on llms.txt versus spending that same day improving the structure and depth of one of your cluster articles, the article is very likely the better investment based on what we currently know.
Site Speed, Mobile Experience, and HTTPS – The Boring Fundamentals That Still Matter
I am going to keep this section relatively short, not because these things do not matter, but because if you have done any serious SEO work in the past several years, you have likely already addressed most of this. The point is simply that none of it has become less important in the AI era – if anything, it has become slightly more important.
Page speed affects how thoroughly and how frequently AI crawlers can access your site within whatever crawl budget they allocate to you. A slow site gets crawled less completely. HTTPS is treated as a baseline trust signal by AI platforms, the same way it has been for search engines for years – if your site is still running on unencrypted HTTP in 2026, that is a problem well beyond GEO. And mobile experience matters because an increasing share of the queries that eventually surface AI-generated answers originate from mobile devices, and AI systems are increasingly factoring mobile-friendliness into how they evaluate sources.
None of this is exciting. All of it is foundational. If your technical SEO has been neglected, technical GEO is not a separate project – it is largely the same project, with a slightly expanded set of bots to think about.
Sitemaps, Internal Linking, and Helping Crawlers Find Your Best Content
A well-maintained XML sitemap, with accurate last-modified dates, helps any crawler – AI or traditional – understand what exists on your site and what has recently changed. This becomes particularly relevant given how much we have discussed the importance of freshness for AI retrieval. If your sitemap accurately reflects when content was substantively updated, you are giving crawlers a direct signal about where the freshness-relevant changes are.
Internal linking does double duty here. We talked in Article 4 about how internal linking helps establish topical authority by showing AI systems that your content exists within a coherent body of work. It also has a more basic technical function: it is literally how crawlers discover pages, especially pages that are not prominently featured in your navigation. A genuinely valuable piece of content that exists as an orphan page – not linked from anywhere else on your site – may simply never be found, no matter how good it is.
Every site has a crawl budget, meaning there is a practical limit to how much of your site any given crawler will explore in a given period. For most small to medium sites this is rarely the binding constraint, but for larger sites – particularly e-commerce sites with thousands of product pages – making sure your most valuable content is not buried many clicks deep in your site architecture, and is referenced from your sitemap with appropriate priority, genuinely affects how completely AI crawlers can explore what you have built.
A Practical Audit You Can Actually Run This Week
Given everything above, here is a realistic, prioritized sequence rather than an overwhelming list of everything you could theoretically do.
Start with robots.txt. Open it, and check explicitly for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and any other AI crawler names you can identify. Confirm none of them are blocked unless you have made a deliberate decision to block them. This takes five minutes and is the highest-leverage check on this entire list.
Next, check your CDN and security settings for bot-management rules that might be catching AI crawlers in a net designed for malicious traffic. If you use a service like Cloudflare, review your bot-fight or bot-management configuration specifically.
Then, view the raw HTML source of your two or three most important pages – the ones you most want AI systems to cite – and confirm your core content is actually present in that raw HTML, not loaded in afterward by JavaScript.
After that, do a spot-check on schema markup
You do not need exhaustive coverage immediately. Prioritize Article schema on your key content pieces, FAQPage schema on pages with genuine FAQ sections, and Organization schema establishing who you are. Google’s Rich Results Test is a free, simple way to verify your markup is implemented correctly.
If you run a developer-facing product with substantial technical documentation, evaluate whether an llms.txt file makes sense for your specific situation. For most other sites, this can wait.
Finally, confirm your XML sitemap is current, accurately reflects recent updates, and includes your most important content. If you have any genuinely valuable pages that are not linked from anywhere else on your site, fix that.
None of this is glamorous. All of it removes friction between the content you have worked hard to create and the systems that might otherwise cite it.
The Bigger Picture: Technical Work Is the Floor, Not the Strategy
I want to close this article with the same caution I raised at the start, because I think it matters. There has been a wave of GEO advice over the past year that frames technical implementation – schema markup, llms.txt, crawler configuration – as the core of the discipline. Some of this comes from tool vendors who have technical solutions to sell, which is not inherently a problem, but it does shape the emphasis of the advice in predictable ways.
The reality, based on everything we have covered across this series, is that technical GEO is necessary but not sufficient. It removes obstacles. It does not create value on its own. A perfectly configured robots.txt file, comprehensive schema markup, and a pristine llms.txt file will not make thin, generic content suddenly become citation-worthy. What they will do is make sure that when you have built something genuinely good – deep, well-structured, authoritative content backed by real third-party credibility – there is nothing standing between that content and the AI systems that might cite it.
Contributed by GuestPosts.biz