Inference Optimization Is a Full-Stack Opportunity

For the past few years, the AI infrastructure conversation has been dominated by training: bigger models, bigger clusters, more GPUs, more data, larger pre-training runs. That conversation still matters. But as AI moves from experimentation into real production usage, the center of gravity is shifting. Increasingly, the bottleneck is inference.

I recently moderated a panel on exactly this at AI-tonomy Summit 2026, with panelists from across the stack: Qi Jin of Cerebras (hardware and software co-design), Nikola Borisov of DeepInfra (production serving and inference clouds), Eugene Cheah of Featherless AI (open-model access), Ash Lewis of Fastino (task-specific models), and Denise Tang of Gradient Ventures (the investor lens). You can watch the full conversation here.

Two takeaways stood out to me. First, optimization actually matters now. For a while, many companies were effectively “tokenmaxxing”: burning tokens because AI was novel, powerful, and suddenly available, not because those tokens were creating proportional productivity gains. That era is ending. Second, inference optimization is not one problem with one fix. It’s a full-stack problem, and a full-stack opportunity. Silicon, serving software, model selection, and the shape of the workload itself all come into play, and you can’t optimize any single layer in isolation.

Why Inference, and Why Now?

Part of the answer is simple adoption: the models got good enough to power real use cases, and those use cases are moving into production. But the more interesting shift is structural. Adoption is up, but it’s the shape of the usage that is transformational.

Denise described it as a chain reaction: “You have more agents, more inference calls per prompt, and more tokens per inference call.” A single prompt now triggers multi-agent orchestration and tool calls behind the scenes; long-horizon agents run for minutes or hours; context windows keep growing. Multiply those factors together and token consumption compounds fast.

Ash put it in terms most developers will recognize: “Back when we ran a prompt into GitHub Copilot, it gave us a single response. Now, if we run a prompt into Claude Code, it’s hundreds of thousands of tokens.” Nikola went further: “If you ask Claude to do something, there might be five million tokens happening in the background in a loop.” And he was blunt about how recently the threshold was crossed: “Six months ago you would still use autocomplete. Now even the best software engineers almost don’t write any code anymore.”

Denise added a point I think is underappreciated: training and inference are converging. In the post-training era, RL rollouts “are essentially mini-inference,” and reward models generate the eval data that feeds the pipeline. Inference capacity is no longer just how you serve a model, but increasingly how you improve one.

A chatbot answering a one-shot question and an agent reasoning through a multi-step workflow are different infrastructure problems. The second creates demands around latency, reliability, memory, throughput, and orchestration that the first never did. That’s why inference has moved to the center of the conversation.

The Cost Paradox

The cost of intelligence has fallen dramatically, and enterprise AI bills are rising anyway.

The numbers on the first half are striking. Serving tokens from a model of equivalent capability costs something like 1,000x less than it did a couple of years ago. Nikola made it concrete: Llama 2, the first good open-source model with a commercial license, cost on the order of a dollar per million tokens. Today, much better models serve for five to ten cents per million, with cached tokens cheaper still.

And yet finance teams are noticing the second half. Enterprises are blowing through their AI budgets: Uber exhausted its entire 2026 allocation in four months and now caps employees at $1,500 a month per coding tool.

Ash shared the data point that best captures the shift: “For the last couple of years, we’ve asked our customers what they care about most when buying models. The answer was always accuracy, followed by latency, followed by cost. Literally in the last two months, that completely changed. Cost has become a factor.” His explanation: models finally got good enough that buyers can afford to care about price.

So the paradox is not really a paradox. To use Nikola’s explanation: “The cost goes down, but because it’s helpful, we use more of it. Sometimes we might overdo it. I think we’re going to adjust, but it’s normal.”

Part of that consumption is productive; AI is being used more because it is useful. Part of it is waste, tokens burned without better outcomes. As budgets become material, discipline will follow, and I expect pricing to evolve with it. Inference is priced per token today, but over time more attention should move to cost per task, per workflow, or per business outcome. Paying for tokens is paying for usage of a tool. Paying for a completed task is much closer to paying for output.

Are We Over-Provisioning Intelligence?

The liveliest disagreement on the panel was about model size, and, tellingly, it mapped almost perfectly onto where each panelist sits in the stack.

Eugene made the case for small models, asking: “Do you want to use an Einstein to summarize your email?” Frontier models will absolutely be used for the hardest problems, he conceded. But “what people want for the vast majority of day-to-day tasks is actually just reliability: the model to get that simple task done, more consistently.” Featherless hosts roughly 40,000 open models precisely because demand is fragmenting toward specialized tasks and languages, and the pattern he sees is companies fine-tuning models in the roughly-30B-parameter class for a single task, then running “billions or even trillions of tokens” through that one task. “It’s about unit economics at scale.”

Ash has lived that arc. His previous company’s dev-agent product, DevGPT, went viral and was spending about $2 million a month on OpenAI inference, all before GPT-4 even launched. Fine-tuning a set of small models brought the bill down to about $20,000 a month, a 100x reduction. His read on the market today: “Really good enterprises already do this. They start with a large model, and then slowly they fine-tune small models and route inference toward them.” The catch is friction: “No one wakes up in the morning thinking, I really want to fine-tune a model today.” It takes strong ML teams, which is why most companies still don’t do it. Even without fine-tuning, model selection alone moves the needle: Kimi K2, he noted, is roughly 75x cheaper than Opus despite being about a third of the size.

Qi took the other side and was refreshingly candid about his incentives: “As hardware guys, we obviously want people to burn more tokens. I’m biased.” But his argument deserves the hearing. In his experience, buyers’ revealed priorities run “intelligence first, capacity second, cost last.” Customers who are getting real value ask for more capability and more capacity long before they ask about price. Small models “work in some narrow situations,” but the highest-value tasks are “all about judgment in a very complex environment that you cannot get wrong.” And in his view we are early on the curve: today’s frontier models have a few trillion parameters, while the human brain has on the order of 100 trillion synapses. “If you get to 100 trillion, then we’re talking.”

Here is how I reconcile them: they are answering different questions. Qi is describing the frontier, the expanding set of judgment-heavy tasks where capability is the binding constraint and the biggest model earns its cost. Eugene and Ash are describing the volume, the well-defined, high-frequency tasks that will make up the bulk of tokens once AI is embedded in production workflows. Both can be right at the same time, which is exactly why routing between them is becoming its own discipline as evidenced by players such as OpenRouter and Sakana’s Fugu system.

The behavioral problem is that teams default to the biggest model because it feels safe. Nobody gets blamed for choosing the frontier model. It’s the “nobody ever got fired for buying IBM” of this era. But as Ash’s numbers show, that instinct can be two orders of magnitude more expensive than the right-sized alternative. The best system will not be the one that uses the biggest model. It will be the one that uses the right model, on the right infrastructure, for the right workload.

Does the Hardware Underneath Matter?

Nikola offered a clean explanation for why inference is a hardware problem at all. Generating a single token from a hundred-billion-parameter model takes hundreds of billions of mathematical operations. Yet fetching a row from a database takes maybe ten thousand CPU instructions. AI requests are millions of times more computationally intensive than the web workloads our infrastructure grew up serving, which is why data centers, accelerators, and electricity keep forcing their way into what looks like a software conversation.

Whether a given buyer should care depends on who they are. Denise drew the line at the persona: agent builders care about model routing, harnesses, and provider economics, not the silicon underneath. Infrastructure engineers at inference clouds and full-stack enterprises care intensely, because their job is securing scarce hardware and then squeezing it with distillation, kernel and compiler work, and serving-layer optimization.

The end state, Qi argued, is abstraction: “At some point you don’t need to care. You just say: I need X gigabytes of intelligence, please give it to me, I’ll pay you.” The technical direction supports him. His own answer to my opening lightning round was hardware-software co-design: GPUs are strong at prefill, wafer-scale systems like Cerebras’ excel at decode, and a software layer that mixes them gets the best of both.

Eugene had the most striking evidence that customers already behave this way. When Featherless launched, its pitch led with the technology: powered by the RWKV linear-attention architecture, heavily optimized, 10x cheaper. “It actually lowered the conversion rate.” The version of the site that converts goes straight to the point: here is the model, here is the price, here is the token speed. “As much as I want people to care about getting rid of transformer attention, most of my customers don’t. They see the price, they get what they want.”

For buyers, that collapses to three numbers and the trade-offs among them. Cost, at the usage you actually expect. Latency, in the dimension your application feels (time to first token for a voice agent, total completion time for a background research workflow). And reliability: uptime, SLAs, and predictable performance, which matters more every quarter as AI becomes operational infrastructure rather than an experiment. The fastest option is rarely the cheapest, and the cheapest is rarely the most reliable. The job is knowing which trade-off your use case can afford.

The Physical Bottlenecks

The part of the discussion that would most surprise people who only follow the model layer: the binding constraints right now are physical.

Memory, not GPUs, was the unpleasant surprise of the last six months. “Sometimes the manufacturers have the GPUs and they’re waiting for memory and drives and CPUs,” Nikola said. “It’s not really Nvidia, it’s all the other components.” Nvidia locking up large memory supply squeezed availability for everyone else and drove whole-system costs up, a part of why Intel and Micron, not just Nvidia, have seen their stocks skyrocket this year.

Qi’s supply-chain picture was bleaker still. From wafers and packaging to networking gear, transformers, cooling, and site permits, everything is running at nine-to-twelve-month lead times. “If you haven’t got your data center right now, you’re talking 2028. 2027 is gone.” His explanation for why supply has not responded is partly psychological: demand arrived all at once, and suppliers do not yet trust it. “Everyone’s a little bit worried, holy smokes, maybe it’s a dot-com bubble, let’s not build it.” His forecast: supply-constrained through 2029 or 2030, followed, if the economic value keeps materializing, by some overbuilding.

Then there is power. “The price of electricity is like gravity for AI,” Nikola said. “If it’s too high, you basically can’t grow in certain places.” Add siting constraints, multi-year construction timelines, and increasingly negative public sentiment toward data centers, and the conclusion is hard to avoid: inference economics are a function of atoms, permits, and watts, not just software.

Where Value Accrues

Denise’s answer was a barbell: the top and the bottom of the stack. At the top, the agent layer accumulates usage and feedback data, which compounds. It improves both the harness and the underlying models, delivering a better experience at lower cost over time. At the bottom, scarcity rules: anyone who can expand or better utilize the supply of chips and data centers captures value for as long as the shortage lasts. Notably, the major model companies increasingly own the agent layer themselves: ChatGPT, Claude, and Claude Code are application businesses sitting on top of models, which is a big part of why those companies do not trade like pure infrastructure providers.

Nikola’s near-term take was simpler: “In an environment like this, where we’ve discovered something really useful, value gets accrued in all the layers. For a while, it’s good everywhere.”

My own view is about sequencing. Infrastructure accrues value first, because it has to exist before the applications can. Over time, value migrates up toward the applications, where workflows are owned, distribution is built, and users experience the value directly. Both things are true; they are just true at different points in the cycle.

Eugene reframed the whole question with a statistic that drives home the macro dynamic: “What astonishes me is that less than 5% of the world is daily-driving AI. My mom and dad barely touch it.” The unlock, in his view, is consumer-grade reliability: “99.999,” trust-it-with-anything dependability for everyday tasks. “Once you have that, whatever inference numbers you’re seeing today, it will explode.” If he is right, today’s supply constraints are a preview, not a peak.

What Will Look Naïve in Two Years

We ran out of time before I could ask the panel my final question: how are we running inference today that will look naïve or obsolete in two years? So I will take a stab.

My answer: the assumption that inference will remain as dependent on one dominant hardware ecosystem as it is today. Nvidia’s full-stack position across silicon, networking, and software is real and durable. But the ingredients of a more heterogeneous future were all sitting on this panel: disaggregated serving that splits prefill and decode across different silicon; wafer-scale systems, TPUs, Trainium, and AMD accelerators finding workload-specific niches; alternative architectures like RWKV attacking the cost curve from the model side; custom, per-model inference engines emerging in the software layer. Even this year’s memory shortage argues the point: concentration in one supply chain is exactly what turns a component squeeze into an industry-wide cost spike.

What is undeniable is that we are early. Usage is growing too fast and the requirements are hardening too quickly for today’s serving patterns to survive unchanged. For founders, that should be encouraging. The headlines chase the biggest models and the biggest clusters, but much of the important work is happening in the less flashy parts of the stack: serving, routing, reliability, model efficiency, workload-specific optimization.

Ash closed the panel with the point that is easy to lose in the infrastructure weeds: “Ultimately, the value here is that we can reduce people’s cognitive load: people have to do fewer of the jobs they don’t want to do. Hopefully we still get to do the jobs we like to do.” Getting there is, increasingly, an inference problem. The cost of intelligence will keep falling. The cost of deploying intelligence will depend on how thoughtfully we build and operate the systems around it.