AI Compute, Scarcity, and the New Infrastructure Stack

For the last two years, the AI conversation has been dominated by models: bigger models, better models, open models, closed models, reasoning models, coding models, and multimodal models. But the more interesting shift may now be happening one layer lower, where model progress runs into the physical limits of infrastructure.

Every prompt, code generation task, agent workflow, video model, enterprise copilot, and inference loop eventually depends on real-world constraints: power, land, memory, packaging, optics, cooling, labor, supply chain allocation, and data center readiness. AI feels abstract because we experience it as tokens — the units of text, data, and context that models process and generate. But every token is produced by physical infrastructure: silicon, copper, concrete, skilled labor, and energy.

That gap matters because demand is compounding faster than the physical world can respond. Token consumption is exploding. Enterprise adoption is still early. Agentic workflows are making each user more compute-intensive. And the infrastructure needed to serve this demand is not elastic. You cannot spin up a gigawatt-scale data center the way you spin up a SaaS instance.

Behind every token is a capital-intensive supply chain: semiconductor equipment, fabs, advanced packaging, HBM, DRAM, substrates, optics, cooling systems, power systems, land, construction, transformers, electricians, HVAC specialists, and data center operators. Some of these constraints are obvious. Everyone now talks about GPUs, HBM, and power. But the less visible bottlenecks may end up mattering just as much: specialty materials, optical subcomponents, underfills, packaging inputs, and regional labor availability.

The problem is not one bottleneck. It is the whack-a-mole nature of the bottlenecks. Solve GPU supply, then hit HBM. Solve HBM, then hit power. Solve power, then hit skilled labor. Solve labor, then discover that a specialty material is supplied by one company that cannot scale capacity quickly enough.

This is what makes AI infrastructure different from prior cloud cycles. Demand can grow with software-like speed, but the supply side still has to move through heavy industrial systems: fabs, grid interconnects, permitting, construction, and manufacturing capacity.

The next phase of AI will likely be shaped less by who can demo the best model in isolation and more by who can turn constrained infrastructure into useful, economically valuable tokens.

From Model Performance to Token Economics

The AI stack is being repriced around a different primitive: the useful token.

Not all tokens are equal. A token used to draft a throwaway email is not economically equivalent to a token that reduces chip design iterations, automates a compliance workflow, or helps an enterprise avoid a seven-figure operational bottleneck. In a world of unconstrained supply, that distinction matters less. In a constrained world, it becomes central.

The market will move from asking how many tokens a company consumes to asking what value those tokens produce. That changes the investment lens. The most important companies may not simply be the ones consuming the most compute. They may be the ones that make each unit of compute more productive.

That productivity can improve at multiple layers of the stack, starting with the business workflow and moving down to the physical infrastructure:

• At the application layer, the question is whether token consumption maps to measurable business value: fewer engineering cycles, higher throughput, lower support cost, faster compliance review, or a capability the customer could not previously deploy.
• At the model layer, smaller models, routing, distillation, mixture-of-experts, and improved reasoning efficiency can reduce the compute required for a given task.
• At the inference software layer, performance depends on how efficiently models are served: batching, routing, caching, speculative decoding, kernel optimization, compiler optimization, autoscaling, and workload placement.
• At the systems layer, memory hierarchy, optical interconnects, co-packaged optics, silicon photonics, data movement, advanced packaging, and accelerator design determine how much useful work can be performed inside a fixed power envelope.
• At the infrastructure layer, liquid cooling, power optimization, data center orchestration, capacity planning, and supply-chain-aware deployment determine whether capacity can actually come online.

The market has been focused on capability. The next phase will care much more about efficiency, and efficiency will be measured across the full stack rather than at any single layer. This is also how we think about our AI portfolio more broadly. Glean sits closer to the enterprise application layer, where the question is whether AI can turn institutional knowledge into measurable workflow productivity. Sakana AI is closer to the model layer, exploring new approaches to building and improving models. We have also made an investment around the inference software layer, where the focus is improving the speed, cost, and reliability of model serving across changing workloads. Smack Technologies applies AI to defense and national security workflows, where the value of better intelligence and faster decision-making is clear. Mueon is deeper in the physical systems layer, where memory integration, packaging, power delivery, thermals, and rack-level architecture become part of the constraint. Different layers, same underlying problem: improving the economics and usefulness of AI.

Why Inference Becomes the Main Event

Training has received most of the attention because it is easier to see: giant clusters, frontier model races, huge capex announcements, and periodic step-changes in model capability. But the longer-term economic pressure is likely to show up more consistently in inference because inference is tied to everyday usage.

A model may be trained periodically, but once deployed, it has to serve every user, workflow, agent, enterprise query, and automated task. As AI moves from consumer experimentation into enterprise workflows, inference becomes the recurring load on the system.

The enterprise shift makes this more important. Hyperscalers and frontier model labs can absorb massive infrastructure costs because they are playing for platform control. Most enterprises cannot. They need AI to translate into productivity, revenue, cost savings, speed, quality, or risk reduction. They will not tolerate indefinite token burn without a path to ROI.

That is why inference efficiency matters. If one architecture can generate two, three, or five times more useful tokens within the same power envelope, it changes what can be deployed economically. A workload that looks too expensive under one cost structure may become practical under another.

But inference efficiency will not be solved by one component alone. Performance increasingly comes from the interaction between the model architecture, the kernel layer, the compiler, the serving runtime, the scheduling system, the memory system, and the data center topology. A 10–20% gain at the kernel layer, another gain from speculative decoding, and another gain from better routing, batching, cache management, autoscaling, and platform orchestration can add up to a very different cost-performance position.

The inference layer may remain dynamic for longer than people expect. The workload mix is changing quickly: coding agents, tool-use agents, customer support, voice and speech pipelines, image and video generation, long-context workflows, and generic containerized workloads where customers want the platform to optimize without hand-tuning every model. Some are latency-sensitive. Some are throughput-sensitive. Some are memory-bound. Some are dominated by long-context prefill. Some are dominated by decode. New hardware, new model architectures, and new workload patterns will keep reopening the optimization problem.

The same pattern is starting to appear around memory. As agents and copilots become more persistent, the issue is no longer just how large a context window can get; it is whether the system can recall the right prior work without turning every interaction into another search problem. Today, long-term memory is often implemented through RAG, agentic search, connectors, or larger context windows. That may work for explicit questions, but it is less suited to workflows where relevant context should surface in real time. If large memory models become part of the serving architecture, rather than just a longer prompt or a search call, memory becomes another lever on token economics: less context reloading, fewer redundant retrieval steps, lower latency, and more useful agents.

What This Means for Founders and Operators

The implication for founders is straightforward: compute is no longer a free background assumption. Scarcity will force customers, investors, and infrastructure providers to ask harder questions about what the product does with every unit of compute it consumes.

For AI application companies, the question is whether token usage maps to a measurable customer outcome. Does the product reduce engineering cycles, automate regulated work, increase throughput, lower support cost, improve conversion, or create a capability the customer could not previously deploy? If the answer is vague, the product is vulnerable. In a constrained market, “we use AI” is not enough. The product has to justify its compute bill.

For infrastructure companies, the question is whether the product expands usable capacity or just moves the bottleneck somewhere else. A founder needs to understand the full dependency chain: wafers, packaging, memory, power, cooling, materials, manufacturing yield, and deployment complexity. A component that looks good in isolation may still fail if it depends on scarce inputs, creates new integration burden, or cannot be deployed inside real data center constraints.

For system-level companies, the test is harsher. The market does not need another component that someone else has to stitch together. It needs architectures that improve the economics of the full deployment. A new accelerator, interconnect, packaging method, cooling system, or power-delivery architecture matters only if it changes useful compute per watt, per dollar, per square foot, or per unit of operational complexity.

The diligence question is whether the company improves the system, or only the slide-level benchmark. Benchmarks matter, but they decay quickly as models, hardware, and workloads change. The more durable question is whether the company can repeatedly turn technical improvement into deployable infrastructure: something that yields, cools, powers, integrates, and operates at scale.

The best companies in this cycle will not simply consume more compute or show better isolated performance. They will make scarce compute more productive.

What I’m Watching

The areas I am most interested in are the ones that make scarce compute more productive, relieve a physical bottleneck, or help customers allocate AI infrastructure more intelligently:

• Inference efficiency / inference operating systems: routing, batching, caching, compiler optimization, autoscaling, hardware placement, and SLA management that lower cost per useful token. 
• Long-context and memory infrastructure: KV-cache management, prefill/decode disaggregation, memory hierarchy, and context-aware routing, and large memory models that reduce repeated search or context reconstruction as context windows expand and agents become more persistent.
• Data movement: optical interconnects, silicon photonics, co-packaged optics, memory-compute integration, and architectures that reduce the cost of moving data across chips, servers, and racks.
• Power, cooling, and deployment: liquid cooling, power delivery, grid interconnect, thermal management, and data center orchestration that determine whether AI capacity can actually come online. 
• Supply chain and capacity intelligence: tools that map bottlenecks across wafers, packaging, memory, substrates, specialty materials, optics, labor, and site readiness, while helping customers forecast and allocate compute.
• Vertical AI with obvious token ROI: engineering, EDA, compliance, healthcare, industrial automation, financial services, and other workflows where time, accuracy, throughput, or risk reduction clearly justify the compute bill.

The Punchline

We are still early in AI adoption, but we are no longer early in discovering the constraints.

The first wave of AI was about capability. The next wave will be about allocation: who gets compute, who can power it, who can deploy it, who can use it efficiently, and who can turn tokens into measurable economic value.

The market will not reward token consumption indefinitely. Scarcity will force discipline. The companies that win will either produce more useful tokens per unit of infrastructure, make the infrastructure easier to build, or apply tokens to workflows where the value is obvious.

The technical and physical frontiers will remain unstable for longer than most people think. New models, new hardware, new context lengths, new agentic workloads, new serving patterns, and new data center architectures will keep reopening the optimization problem. That means the winners may not be the companies with the best static benchmark today, but the companies that can keep redefining the cost-performance frontier as the workload mix changes.

AI may look like software to the user. But the next trillion dollars of value will be fought over in the physical and systems stack underneath it.