The Search for AI’s CDN

Artificial intelligence keeps re-thinking thoughts it has already had. Whoever fixes that will own the next great infrastructure layer.

Somewhere right now, an AI agent is reading a 200-page policy manual for the thousandth time today—re-processed from scratch, at full price, because the system serving it has no memory of having read it before. Multiply across every agent, enterprise, and request, and you have the defining waste of the inference era.

The web solved its version of this a generation ago: Akamai, Cloudflare, and Fastly built a multi-billion-dollar layer on the observation that most requests are redundant, so the answer should be cached close to demand. AI inference is now rediscovering that logic—except the asset worth caching is not content. It is computation.

I believe it is a market-structure question, not an optimization detail. Whoever defines the caching layer for inference will occupy the position content-delivery networks held for the web: invisible, indispensable, and tolled on every request.

Why Inference Needs a CDN Now

A Content Delivery Network (CDN) caches static content—HTML, images, video—on edge servers geographically distributed close to end users, so requests resolve locally instead of round-tripping to a single origin server, cutting latency and origin load.

Three forces are converging.

Redundancy at scale. An enterprise running thousands of agents recomputes the same things thousands of times: shared system prompts, tool catalogs, policy documents, codebases. One industry analysis found roughly 31% of LLM queries are semantically similar to earlier ones—before counting exact repetition, which runs far higher in agent workflows.

The economics are already priced in. The frontier labs have quietly conceded that recalled computation is a different commodity from fresh. Anthropic bills cached input at 10% of its standard rate, and OpenAI’s newest models match that spread—$0.50 per million tokens cached against $5 fresh (mid-2026 pricing). A tenfold price gap between “computed” and “remembered” is not a promotion. I think it is the market marking where the next infrastructure layer sits.

Agents made inference stateful. The chatbot years, 2022 to 2024, were stateless: request in, response out, nothing retained. Agents invert that. They hold context for hours, call models hundreds of times, and coordinate with other agents—accumulating state that is expensive to rebuild and valuable to keep. Stateless infrastructure serving stateful workloads is the mismatch this debate exists to resolve.

Why the Original CDNs Can’t Simply Extend to This

The obvious objection: Cloudflare and Akamai already operate thousands of points of presence. Why isn’t AI’s CDN simply their next product line?

Because the asset breaks every assumption the web CDN was built on. Consider the leading candidate for that asset: the KV cache¹, a model’s stored working memory of a prompt.

A web asset is universal; AI state is not. A JPEG is identical for every user. A KV cache is specific to one model, one version, one way of encoding text—cache it for Llama and it is useless to Claude. The web CDN served one copy to a billion users; the AI cache serves fragments to exactly-matching requests.

A web cache lives apart from compute; an AI cache is welded to it. Edge caching worked because a cheap server with a disk could serve the asset. A KV cache is useful only beside a GPU running that exact model; without the compute, it is inert bytes.

The sizes are inverted. Web assets are kilobytes; a 100,000-token context on a large model can exceed 100 gigabytes of state. Placement, not replication, becomes the core discipline.

The edge pioneers see this. Cloudflare is installing GPUs across its 330-plus-city network and rebuilding itself as a distributed inference platform; Akamai is pursuing the same pivot. Note what they are building, though: compute-plus-state infrastructure, not asset replication. The old form does not transfer; even its inventors are constructing a new one.

A deeper shift makes this urgent: inference itself is decentralizing and diversifying. The first AI wave was maximally centralized: a handful of hyperscale regions, frontier models behind single APIs. It is unwinding under sovereign-AI mandates, data-residency rules, and GPU capacity spreading across specialist “neocloud” providers, on-premise clusters, and the edge. A centralized web never needed a CDN; a global one did. Once an agent’s state must follow it across regions, providers, and hardware, inference cannot function without a layer that answers one question: where does the intelligence already live, and how do we get the request to it?

Five Candidates for the Job

If AI needs a CDN, what exactly gets cached? I thought through five candidates, in ascending order of seriousness

1. Model weights.

• The case: the most literal analog—replicate the model near demand, exactly as CDNs replicated files. 
• The counter: weights are static between releases, and distributing them is a solved problem—registries² and regional deployment³, done once, not per-request. There is no recurring economic event to monetize. 
• Verdict: weights are the single origin server, not the cache.

2. Semantic response caches—store full answers and serve them when a new query is “close enough.” 

• The case: when it hits, the savings are total, since the entire model call is eliminated, and that 31% similarity figure suggests a large surface. 
• The counter: “close enough” is a probabilistic judgment, and a CDN cannot be probabilistic about correctness. Personalization, freshness, and compliance all break reuse; one confidently wrong recycled answer in a regulated workflow erases a year of savings. 
• Verdict: It sounds like a legitimate application-layer feature, not the infrastructure layer. The web CDN never guessed whether your page was similar enough to someone else’s.

3. Model routers and AI gateways—OpenRouter, Vercel’s AI Gateway, and most recently Ramp Router: one endpoint that picks the best model and provider for each request. This one deserves a hard look; the resemblance is uncanny. Routers hold the CDN’s exact topological position—the layer between demand and origin, touching every request—where Akamai and Cloudflare themselves began, as thin proxies and overlay networks. Ramp’s router learns provider latency and failure rates in real time to pick the cheapest reliable route—roughly 30% savings across trillions of daily tokens (company-reported, July 2026)—the CDN request-routing problem of 1999, solved with modern statistics. 

• Why I set it aside—for now: the CDN’s defining act is serving from cache instead of origin, and a router sends every request to origin. It makes each request cheaper by shopping, but it never makes the second request cheaper than the first. Worse, cost-optimal routing can be actively cache-destructive. Provider-side caches are prefix-specific, so bouncing an agent’s fifth call to whoever is momentarily cheapest forfeits the warm cache and pays for a full re-read. Per-request optimization becomes per-session pessimization. 
• Verdict: I think routers are building the control plane of AI’s CDN without its data plane—the DNS and load balancer of the AI era, not its cache. The moment one starts routing on state locality—sending requests where their computed context already sits—it converts from proxy to proto-CDN. It has not happened yet and may never.

4. Retrieval and context stores—the vector databases and context platforms holding the documents and memory fed into models. 

• The case: the strongest challenger on durability. Context is model-agnostic, so it survives the one event that kills every other candidate: a model upgrade. Swap models, and the context store persists while every cached computation evaporates. 
• The counter: durability is not the CDN’s job—acceleration is. The context store stops at the model boundary, so every request still pays full prefill⁴. Caching the library does not save you the reading. 
• Verdict: I believe, the context store is the system of record. The CDN question is about the system of recall.

5. The KV cache.

• The case: prefill is the single most expensive, most redundant step in inference, and the KV cache stores its output. Cache it once, and every later request over the same context skips straight to generation. It is the only candidate that caches computation itself, at exact fidelity, inside the hot path—where the FLOPs, and therefore the dollars, concentrate. 
• The counter—and it is real: this is a fragile asset. Every model release I think invalidates the entire cache, and a CDN whose inventory expires on a vendor’s release schedule is a strange CDN. Exact-prefix matching⁵ is brittle: change one early token and everything downstream is void. And the assets are enormous, which makes distribution itself costly. 
• My response: fragility is a renewal cycle, not a disqualifier—web caches expired too, just on gentler clocks—and agent workloads are precisely the ones with the long, stable, shared prefixes that caching rewards. The mechanics, meanwhile, are being solved in public. LMCache, an open-source engine for storing and reusing KV state across machines, has become the caching layer inside NVIDIA’s Dynamo and vLLM’s production stack. Moonshot’s Mooncake, which serves the Kimi assistant at consumer scale, is architected explicitly to trade cheap storage for expensive computation. VAST Data reports a twentyfold improvement in time-to-first-token from cache offload (vendor-reported, Dec 2025). 
• Verdict: the KV cache is the cached asset in CDN. And with NVIDIA researchers actually demonstrating cache transfer (Aug 2026) across AI models, it now truly has the potential to serve as “AI’s CDN”. I hypothesize that the durable value shall sit not just in the cache but in the orchestration above it.

The Architecture That Follows

It will resolve as every caching system has: in tiers. The GPU’s own onboard memory holds actively generating state, fastest and scarcest. Pooled memory shared across a cluster, enabled by new interconnects like CXL⁶, holds warm state for the neighborhood. SSDs and object storage hold the cold state of sleeping agents. Above it all sits a new class of inference router computing a live equation on every request: queue for the GPU that already holds this state, or grab an idle GPU and pay the transfer toll? Routing on model, hardware, state locality, queue depth, network, and cost—this is the load balancer of the AI era, and gateway players and datacenter schedulers are converging on it from opposite ends of the stack.

What This Means for Founders and Operators

If you’re building infrastructure: Don’t build just the cache. Build the orchestrating router that knows where the cache already lives. Storage is commodity and state expires fast—the money is in deciding what to place, where, and when to throw it away.

If you’re deploying agents: cache-hit rate is becoming a P&L line. By some FinOps estimates, teams leave 70–90% of input-token spend on the table through poor prompt structure. Put stable content first, keep session affinity to one provider where caches are warm, and trach hit rates the way you once tracked CDN offload.

I believe the CDN market rewarded neither origin servers nor browsers—it rewarded the layer between. The winner here is likely to emerge from a router acquiring state-awareness than from a storage layer trying to route demand, because in infrastructure, the layer that owns the request decides where everything else lives.

The Punchline

For content distribution, the web largely needed a CDN once it went global. Inference only needs one now that it is going distributed and diversifying in usage—and it is. The web’s CDN cached what everyone could see; AI’s CDN will cache what the machine has already thought and indexed based on requests. 

The companies that make intelligence recallable instead of recomputable will set the unit economics of the inference era.

If you are a founder building at this layer, I would love to hear from you.

——————————————

¹KV cache (key-value cache): the stored intermediate computation a model produces while reading a context—its “working memory” of a prompt, reusable so the reading isn’t repeated.

²Registry is a centralized, secure storage repository used to host, version, and manage large digital assets.

³Regional deployment means copying and storing those model weights in multiple physical data centers (regions) around the world closer to the end-users.

⁴Prefill: the expensive first step of inference, where the model reads and encodes the entire context before generating its first word.

⁵Prefix matching: caches apply only when a new prompt begins with the exact same token sequence as a cached one, on the same provider and model.

⁶CXL (Compute Express Link): an emerging interconnect standard letting many processors share a common memory pool at near-local speed.