Teardown No. 07 · llms.txt

How Umar Gora tested whether AI crawlers read llms.txt

Umar Gora checked 63 days of IP-verified server logs. Seven AI and search agents made 2,456 verified requests to his site, and none asked for /llms.txt or /llms-full.txt. The same agents read the markdown "twins" of his posts 75 times.

Umar Gora · @UmarrGoraVerified Oct 13, 2026Markdown copy

The problem

Everyone recommends llms.txt, but few people check their logs to see whether anything requests it.

The blueprint

  1. Publish a markdown twin of every post at the same path with .md added.
  2. Link the twin from each HTML page’s head with rel="alternate". llms.txt also lists every twin.
  3. Verify crawler IPs before counting a request, so only verified ClaudeBot, GPTBot, OAI-SearchBot and similar requests count.
  4. Tabulate per agent: requests for robots.txt, the sitemap, markdown twins and llms.txt.
  5. Act on the result. He would build twins and the rel="alternate" link first and treat the root index file as optional.

Stated results

Over 63 days: 2,456 verified requests from 7 agents, with 0 for /llms.txt or /llms-full.txt. Markdown twins got 75 verified reads: ClaudeBot 31, GPTBot 29, OAI-SearchBot 9, Bingbot 6. The twins had 144 requests in total across 31 pages.

Why this matters for postby.ai

  • Ship llms.txt, because it’s cheap.
  • Also ship markdown twins and rel="alternate" links for every teardown.
  • Measure in our own logs rather than assuming it works.

Caveats

  • X: @UmarrGora (verified 2026-10-13: linked from umar.codes).
  • One site’s logs. Results may differ on larger sites.

Sources

Get one teardown a week.

Same format, a new operator every issue. Double opt-in.