Why does token count matter for a website?

Every page an AI agent fetches and reads consumes tokens from its context window, and those tokens have a real dollar cost if the agent is calling a paid LLM. A homepage bloated with navigation markup, inline scripts, and repeated boilerplate costs an agent far more to read than the actual useful content on the page is worth.

What does the AX Audit measure for Token Efficiency?

Two things: an estimate of how many tokens your homepage costs to read (counting the rendered page content, not raw markup), and whether you publish an llms-full.txt file giving agents a condensed, complete-context alternative to crawling the whole site.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short, curated index — links and a one-line summary per resource. llms-full.txt is the complete content inline in one file: the actual text an agent would otherwise have to crawl page-by-page to assemble, pre-condensed into a single, token-efficient document.

What's ainative.ax's own Token Efficiency score?

ainative.ax currently scores 3 of 4 on this check — the homepage itself runs at roughly 3,461 tokens, and llms-full.txt is published and available. The one point not yet earned reflects the homepage token count still being above the check's full-credit threshold.

← All guides

AR-CONT · CONTENT DELIVERY

Token Efficiency — The Cost of Reading Your Site

An agent reading your homepage pays for every token, whether it's useful content or navigation boilerplate. Token Efficiency measures how much of that cost you're forcing agents to absorb.

Why This Is a Real Cost, Not an Abstraction

When an LLM-backed agent fetches a page, the page’s content becomes part of its context window. If that agent is running on a paid API, every token of navigation markup, repeated footer links, and inline tracking scripts is a token the operator is billing someone for — with zero informational value in return.

At the scale agents increasingly operate at (checking hundreds of sites in a research task, or re-reading a page on every tool call because there’s no cheaper alternative), a bloated homepage compounds into a real, measurable cost disadvantage against a leaner competitor serving the same information.

What the AX Audit Actually Measures

Homepage token count

An estimate of how many tokens the rendered homepage content costs to read — lower is better, and the gap between a lean and bloated homepage is often 10x or more.

llms-full.txt availability

Whether a complete, pre-condensed content file exists so agents can get everything in one request instead of crawling page-by-page.

Both signals point at the same underlying goal: an agent should be able to get the information it needs from your site at the lowest possible token cost, whether that’s by trimming the default page or offering a dedicated condensed alternative.

The Two Ways to Improve This Score

Reduce the homepage itself.Server-render content instead of hydrating it client-side with verbose component markup. Collapse repeated navigation and footer structure. Move large inline scripts and styles to cached external files the token-counter won’t attribute to the page content. None of this changes what a human sees — it changes how much of the response is actual content versus plumbing.

Publish llms-full.txt. Even if your homepage stays as-is for human visitors, a condensed, complete-content file at /llms-full.txtgives agents an explicit, cheap alternative entry point. This is the lower-effort fix — you don’t need to touch your rendering pipeline, just maintain one additional file.

# llms-full.txt — complete context, condensed

## Product
AX Audit scores websites for AI-agent readiness...

## Pricing
Free tier: unlimited quick scans...

## API
POST /api/v1/public/ax-audit
  body: { "url": "..." }
  returns: { score, grade, checks[] }

[... full content, no HTML, no navigation, no scripts ...]

Where This Fits in AR-* and the AX Score

Token Efficiency falls under AR-CONT, the category covering how content is delivered once an agent has found it. In the live AX Audit, it’s its own named check worth up to 4 of the 100 points, split between homepage token count and llms-full.txt availability.

ainative.ax currently scores 3 of 4 on this check: the homepage runs at roughly 3,461 tokens and llms-full.txtis published, but the homepage token count is still above the threshold for full credit. That’s a disclosed, specific gap — not a rounding error hidden in a vague claim.

Frequently Asked Questions

Is there a hard token budget I should target?▼

The audit doesn’t publish a fixed public threshold — it’s a relative measure, and lower is consistently better. As a practical anchor, a homepage well under 5,000 tokens with its substantive content intact is a reasonable target for most marketing or product pages.

Does minifying my HTML/CSS/JS help?▼

Minification reduces bytes transferred but doesn’t meaningfully reduce the token count of the rendered text content an LLM actually reads — token counting works on the semantic content, not raw file size. The bigger win is removing boilerplate markup and repeated structure, not compressing whitespace.

Should llms-full.txt include everything on my site?▼

It should include everything you’d want an agent to know without crawling further — product description, pricing, API reference summary, key policies. It doesn’t need to include every blog post or marginal page; link out to those instead of inlining them.

Check Your Token Efficiency Score

The AX Audit estimates your homepage's token cost and checks whether llms-full.txt gives agents a cheaper alternative.

Check your token efficiency score→

Free audit. No signup required.