Markdown Twins: Serving Content AI Crawlers Can Read

What a markdown twin is

A markdown twin is the plain-text version of a page, served at that page's own URL with .md appended.

Take any article on this site and add the suffix. You get the source text: headings, prose, tables, links. No navigation, no scripts, no layout, no cookie banner. Every twin is listed in llms.txt, a plain manifest at the root of the domain.

The idea is straightforward. A system trying to read your content currently has to fetch HTML, run it through a parser, strip the chrome, and guess at what was content and what was furniture. A twin skips all of that.

What it is honestly worth

I want to set expectations before the build details, because this topic attracts more enthusiasm than it deserves.

No crawler is contractually obliged to read llms.txt. It is a convention, not a standard, and nobody should present it as a ranking factor. If a vendor is selling llms.txt as an AI visibility service, they are selling a text file.

What it does is narrow and real: it removes the parsing step, it publishes a clean index of your content at a predictable address, and it costs close to nothing once it is wired up. On this site the twins are generated during the build, so they add build time and no runtime cost at all.

That is the entire claim. Cheap, tidy, plausibly useful, not a growth lever. Everything past this point is engineering.

How the build works

This site runs Next.js with the App Router. The twins come from one route handler that serves any /articles/... path, with a proxy rewrite mapping /articles/<slug>.md onto it.

The important decision is the boring one: the twins are generated at build time, not per request.

That looks like an optimisation and it is actually a correctness requirement, which brings us to the interesting part of this article.

Every twin on this site returned 404 for three weeks

I sat down to write this piece, decided I should verify the infrastructure before describing it in public, and found that all 226 markdown twins were returning 404 on the live site.

They returned 200 locally. They had returned 200 locally the entire time.

The cause is specific to serverless deployment and worth understanding, because nothing about it is obvious from inside the application. The route was dynamic, so it read article files from disk when a request arrived. Next.js decides which files to bundle with a deployed function by tracing the imports and file references it can see in your code. It cannot follow a path that is assembled at runtime from a URL segment. So the article files were never bundled, the deployed function had nothing to read, and every request 404'd.

Locally, next start serves the real filesystem. The files were right there. That is the entire reason it looked fine for three weeks.

Everything around it worked, which is what hid the fault

This is the part I would want someone else to take from this article.

The proxy rewrite was firing correctly, and I confirmed it. The article pages themselves are statically generated, so they were unaffected. And /llms.txt is a static route, baked at build time, so it returned 200 with a complete list.

So the one file whose entire job is to advertise the twins to crawlers was working perfectly, and it was advertising 226 URLs that all 404'd. Every individual signal I would normally check was green. The site was confidently handing ChatGPT, Claude and Perplexity a list of dead links.

The fix was to make the route static, prerendering every twin during the build, where the files genuinely exist. The build went from 227 prerendered pages to 444.

The first fix was also wrong, and also silent

I generated the parameters for articles and shipped it.

That built cleanly, and it served 44 working twins while 183 category and author URLs kept returning 404, because the manifest advertises all three kinds. A partial fix produced a green build and a mostly-broken feature, exactly like the original bug.

So the real fix was not the code change. It was an automated check: the build now compares the twins advertised in llms.txt against the set actually generated, and fails on any gap. It reports 226 advertised, 226 prerendered. If those numbers ever diverge again, the build stops instead of the site quietly lying to crawlers.

If you build this, build the check with it. The feature has no visible failure mode, which means it needs one.

Doing this on Shopify

Shopify does not let you add arbitrary routes and content types the way a framework does, so the pattern needs adapting rather than copying. Two approaches actually work.

Alternate templates. Shopify can render a URL through a named template variant, and a template that suppresses the theme layout can output stripped-down text instead of a full page. This is the low-effort option: no external infrastructure, and it uses the theme system you already have. The limitation is that you are still being served an HTML response, so it is a plain page rather than a genuine plain-text document.

An app proxy. Shopify forwards requests from a path under your own domain to a server you control, and you decide what comes back. That gives you real control over the response, which makes it the better fit if you want twins that behave properly, and it is more work because it means running something.

For most stores I would start with the first, and only reach for the second if the content library is large enough that AI retrieval is a genuine strategic concern rather than a nice-to-have.

Worth naming the harder truth: on a typical store, the constraint is almost never the format your collection descriptions are served in. It is that the collection descriptions do not exist. Getting the text written matters more than getting it served twice.

The checklist

If you are going to build this, in order:

1. Pick the pages whose value is their text. Articles, guides, collection copy, documentation. Not cart, checkout or account pages.
2. Use a guessable URL pattern. The page's own path plus .md. Predictable beats clever, because anything reading your site can then construct the address without being told.
3. Generate at build time. A twin assembled per request is the exact code path that breaks on deployment.
4. Publish a manifest at /llms.txt.
5. Add a build check that the manifest and the twins agree.
6. Request one against production and look at the status code. Not the dev server. Not the rendered output. The status code, on the deployed site.

Step six is the one I skipped, and it is the reason this article has a war story in the middle of it.

The general lesson

The narrow point is about markdown twins. The wider one is about any infrastructure whose only consumer is a machine.

A feature that people use tells you when it breaks, because they complain. A feature that only crawlers use will fail in production, keep passing every test you have, and cost you nothing you can see until you go looking. The three weeks this site spent serving a directory of dead URLs to AI crawlers cost me nothing measurable, and I only found it because I decided to check a claim before publishing it.

If you have built something in this category, go and request it against the deployed site right now. It takes a minute and I would not bet on the result.

I build content infrastructure like this for e-commerce stores, mostly on Shopify. If that is a problem you have, tell me what you sell. The data behind why any of this matters is in a separate article, with the referral numbers to go with it.