.svg)
There’s been much discussion (and debate) about Markdown for AI search lately. Does it work? Is it a distraction? Should we use it? Should we skip it?
Here at Botify, we’re combining insights from research & testing with our analysis of what’s happening in the industry to provide recommendations on how and whether you should consider optimizing for AI bots using Markdown.
What we know about Markdown
Let’s start with what we know and go from there.
What is Markdown?
Markdown is an incredibly simple markup language for content. It doesn’t rely on complex code to convey formatting and hierarchy. Instead, it uses simple symbols to denote things like headers, links, text emphasis, and more.
This results in significant bandwidth reduction for the systems accessing content: by stripping headers, footers, navigation, and other non-essential elements, it leaves only “hero” content and reduces page weight by 3–5x.
It also results in faster response times, which increase your likelihood of being included in an AI summary. No resource-strapped agent can wait forever, so if after a few hundred milliseconds it doesn’t have what it needs, it may only prioritize and summarize the parts of your content that responded first. That means critical information could get left behind — and if AI can’t see it, it can’t share it with your customers.
Why is Markdown important for AI?
Markdown’s simplicity makes it easy to read and understand information, without the noise of code like HTML. LLMs are, in essence, a sophisticated prediction engine, one that analyzes patterns and context to generate the most likely output. LLMs trawling through pages of code have to sift through an incredible amount of noise to find the information that’s relevant to people, within natural language and context. It takes processing power to parse and understand information, and the AI agents and bots exploring content have finite resources. Providing them content in an easy-to-understand format means they can find more of it faster.
The Markdown controversy
There has been debate around whether supplying content in Markdown to AI bots is worth the effort or not. Google spokespeople like John Mueller have made statements implying that it won’t make a difference, yet we have also seen Google prioritizing Markdown in initiatives like the Open Knowledge Format, where they describe it as “readable in any editor, renderable on GitHub, [and] indexable by any search tool.”
Experimentation & research by Botify
We foster a sense of curiosity here at Botify; it’s our goal to understand both what works today for our customers, and to help them prepare for what will work tomorrow. We have significant data on crawlers and crawler behavior — we know how they like to consume content. We also have some of the most brilliant minds in AI search on our team. That deep, evidence-based knowledge of bot behavior combined with expert analysis has given us special insight into how Markdown is actually being consumed by bots and agents today, with hints at what to expect in the future.
To begin understanding the relationship between Markdown and AI bots, Botify’s own AI search consultant and bot researcher, Mike Levin, devised an experiment using his own website that studies bots as they explore in real time.
Testing methodology
There are four main ways bots find Markdown content in the first place:
- Standard hyperlinks (<a href=...>): The traditional way. A visible link on a page pointing directly to a .md file.
- HTML head discovery (<link rel="alternate" …): This is an invisible tag in the <head> of the HTML document telling bots, "Hey, there’s a cleaner, machine-readable Markdown version of this exact page over here."
- Direct agent maps (llms.txt): LLMs.txt is the AI equivalent of a robots.txt or sitemap.xml. It's a centralized directory that tells AI agents exactly where to find the most important conceptual information.
- Content negotiation (the Accept header): This is the most advanced method. When a bot requests a URL, it sends a hidden list of formats it prefers (the HTTP Accept header). If it says "I prefer text/Markdown," the server intercepts the request and hands back Markdown instead of the standard HTML page.
To trace how bots find, request, and potentially use Markdown on a website, Mike devised a way to track their entry points and format requests. Setting up a self-hosted server, he intentionally hosts his blog on the open internet without a CDN (such as Cloudflare); this means that there’s no layer that filters or absorbs bot traffic before it hits the server, and every AI bot request is captured in the access logs. With this set up, he was able to control and track technical factors, such as Nginx routing, the SQLite telemetry databases, and the exact content delivered to specific user agents.
Let’s dive into what we found.
Are bots actually reading Markdown?
The short answer is yes. AI agents and crawlers are actively seeking out and ingesting raw Markdown. While HTML remains the main format for human traffic, Markdown is rapidly becoming a high-signal blueprint for machine readers.
For those who love to dive into the data details, we can see how discovery is tiered by the types of bots and agents requesting Markdown:
The standard paths, HTML head and standard links, are preferred by the exact agents publishers care about:
- GPTBot (OpenAI): 1,273 combined reads across direct links and HTML head discovery
- ClaudeBot (Anthropic): 615 combined reads
- Facebook / Meta External Agent: 470 combined reads
- Ahrefs & search crawlers: Heavy ingestion, proving that traditional SEO tools are also archiving this format
Meanwhile, the content negotiation path sees the fewest requests. This is likely because most crawlers are defaulting to asking for HTML, which is legacy crawler architecture.
The findings
AI bots act differently + measurement matters
Not all AI bots behave the same way, and volume is the wrong way to measure them. If you just count requests, the biggest crawlers look most important, because of course they make the most requests. But if you instead ask "what percentage of this bot's visits ask for Markdown (the clean text version of a page) instead of HTML," the bots divvy up into clear groups and the narrative changes a bit.
In a nutshell, the bots crawling the most usually don't care about Markdown, and the bots that do want Markdown tend to be small, new, and growing quickly. Request counts show noise, whereas Markdown share indicates which bots are built for an AI-first web.
Who wants Markdown the most?
The answer: purpose-built AI tools want Markdown the most. These are bots and agents that seek “menu” files intended for LLMs (like LLMs.txt), and those who search for clean, machine-readable text to add straight into a model's context. They request it close to 100% of the time, and they're the ones using the content negotiation channel: the smallest column in the table above, but the one occupied almost entirely by the newest, most AI-native traffic.
Who wants Markdown the least?
Legacy crawlers, like traditional backlink / link-graph tools and infrastructure scanners, are the least Markdown-hungry. The reason for this is that Markdown deliberately strips out the things those crawlers look for, such as navigation, anchor tags, and link scaffolding. They prefer raw HTML because Markdown simply won’t give them what they need.
Bots from the same company may differ in Markdown preference
To get a sense of where this is heading, we’re seeing that the divide runs within companies, not just between them. For instance, Anthropic's bulk crawler, ClaudeBot, largely takes whatever HTML it finds; however, Claude Code, the assistant people actually run to get answers, requests Markdown nearly every time.
As agentic assistants become the front door between your brand and your customers, that assistant behavior is the one worth designing for.
What’s the verdict — Markdown or no Markdown?
AI search and agentic commerce are still in their relative infancy, but the industry is innovating at an exponential rate. We know, from testing evidence and from watching what the biggest players are prioritizing, that making site content simple and easy for bots to consume is your fastest track to getting your product and brand data into the systems that make recommendations directly to your customers. Even if Markdown isn’t the top request from AI bots today, this is a future-proof standard to adopt when strategizing content delivery to AI agents; it’s an ideal communication interface for LLMs.
Deploying Markdown is a low-risk, high-signal investment in the future of AI search:
- It provides a stripped-down, noise-free version of site content, removing DOM clutter (navigation, ads) that confuses AI tokenizers
- It utilizes the exact format LLMs are natively trained on and generate
- By implementing llms.txt and <link rel="alternate"> tags, they explicitly guide AI agents to their most valuable content
Botify’s take on Markdown
We believe that for an ideal bot experience, it’s better to serve only the most relevant content by stripping away the template. This prevents AI bots from having to "clean" the page themselves, which could lead to data loss. Consider it like a best practice for optimizing an agent’s experience on your website (in the same realm as implementing the right metadata on-page.)
For high-traffic websites, this also reduces the payload size and speeds up decompression. Consider this: when ChatGPT fetches five pages via web search, it likely operates on a strict timeout (for example, 500 milliseconds to 1 second) to fetch, decompress, and parse the HTML. Serving Markdown to AI bots makes that process seamless and thus stacks the deck in your favor.
How to send Markdown to bots and agents quickly and easily
We know Markdown makes it quick and easy for bots to get the content they need. How hard is it, then, to translate your existing content into this format?
At scale, it’s generally not simple or fast. Enterprise sites may struggle with having the Markdown version available to serve, if requested, and the details of implementing content negotiation per the semantic web specification.
That’s why we’ve added the option to serve Markdown to AI bots through SpeedWorkers, our edge network bot management solution. SpeedWorkers delivers optimized content to bots, accelerating delivery, ensuring visibility, and protecting your website infrastructure. With SpeedWorkers, you can opt to select the AI bots of your choice and determine which format you prefer for SpeedWorkers to serve.
It’s easy to preview what that formatting will look like to QA the bot experience along the way.
SpeedWorkers scales with any enterprise website to ensure that bots are receiving content in the format that they prefer, whether it be HTML, Markdown, or a combination of the two.
While the conversation about Markdown continues, we want to provide content to bots in the format they request. The ultimate goal is to enable bots to easily digest and understand your brand’s content at scale.
Conclusion
If you want to be found in AI search, the Markdown conversation isn’t one to miss. In the end, giving agents the content they’re looking for in the format they prefer is another lever you can pull to boost your visibility with consumers. If you’re curious to learn more, reach out to our team anytime — we’d love to discuss!
.svg)
.svg)
.svg)
