AI is becoming cheaper to run, yet increasingly expensive to operate. The next competitive advantage may not be intelligence alone, but the economics of delivering it.
The Statement That Should Make AI Product Leaders Think
On November 1, 2026, Elon Musk made an interesting admission about Grok Bot.
“The cost of serving Grok Bot was unsustainably high relative to revenue.”
The statement came amid complaints from users who were exhausting their usage allowances sooner than expected.
What makes this interesting isn’t necessarily the company or the product.
It’s the underlying economics.
For years, the technology industry has operated on a relatively straightforward assumption: build a great product, acquire users, scale infrastructure, and eventually improve margins through economies of scale.
Traditional software-as-a-service businesses have often benefited from this model. Once the software exists, serving additional customers can be relatively inexpensive compared with the revenue they generate.
AI complicates that equation.
Every model invocation consumes computation. Every additional reasoning step increases resource utilization. Every autonomous agent potentially introduces multiple model calls, tool executions, context retrieval operations, and retries.
The system doesn’t simply serve the customer.
It performs computational work on the customer’s behalf, sometimes continuously.
And that creates an uncomfortable possibility:
An AI product can grow rapidly, attract millions of users, generate substantial revenue, and still have unsustainable unit economics.
The distinction between adoption and profitability is becoming one of the most important conversations in AI product strategy.
The Research Reveals a Contradiction
Here’s where things become particularly interesting.
AI inference is becoming significantly cheaper.
According to Stanford’s 2025 AI Index Report, the cost of querying a model with performance equivalent to GPT-3.5 on the MMLU benchmark declined from approximately $20 per million tokens in November 2022 to just $0.07 by October 2024.
That’s a reduction of more than 280 times in approximately 18 months.
An extraordinary improvement.
Yet companies continue to face enormous infrastructure and inference expenditure.
How can both things be true?
Because reducing the price of an individual unit of computation doesn’t necessarily reduce the total computation consumed.
As AI becomes more accessible, organizations introduce it into more workflows. Applications become increasingly sophisticated. Models process larger contexts, perform deeper reasoning, and execute increasingly complex tasks.
A chatbot that once generated a single response can evolve into an agent that independently executes dozens of actions.
The cost per token is falling, but the number of tokens required to deliver certain outcomes can increase dramatically.
This distinction matters.
We’re witnessing two competing forces: declining computational prices and expanding computational consumption.
The financial outcome depends on which force dominates, alongside the revenue and value generated.
What Stanford’s Research Tells Us About AI Agents
A 2026 study associated with Stanford’s Digital Economy Lab, titled How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks, investigated how AI agents consume tokens.
The researchers analyzed execution trajectories from eight frontier language models performing software engineering tasks.
Their findings were revealing.
In the workloads studied, agentic tasks consumed approximately 1,000 times more tokens than conventional code reasoning and chat interactions.
Even more interestingly, repeated executions of the same task could differ by as much as 30 times in total token consumption.
And greater token consumption did not consistently produce better outcomes.
In some cases, accuracy peaked at intermediate expenditure levels before additional computation delivered diminishing returns.
The researchers also found that models struggled to predict their own token consumption before performing tasks.
These findings come from agentic coding workloads and should not be interpreted as universal cost multipliers across every industry.
Nevertheless, they highlight a fundamental challenge.
We are increasingly deploying systems whose computational expenditure is difficult to predict, even when the underlying task remains unchanged.
Consider what this means for an enterprise AI product.
A customer might pay a fixed subscription fee.
But the computational effort required to serve that customer can vary substantially depending on how they use the product.
One user submits straightforward questions requiring minimal reasoning.
Another executes complex, multi-step workflows that require repeated retrieval, analysis, validation, and tool invocation.
Both may generate similar subscription revenue.
Yet their serving costs could be dramatically different.
This makes traditional subscription pricing more complicated.
It also introduces a new product management challenge: predicting and controlling the economics of autonomous workflows.
The $600 Billion Question Was Never Just About Infrastructure
In June 2024, Sequoia Capital’s David Cahn published AI’s $600B Question.
The article examined the enormous investments flowing into AI infrastructure and the revenue necessary to economically justify that investment.
The $600 billion figure represented an illustrative annual revenue requirement implied by infrastructure spending assumptions, rather than an observed industry loss.
The underlying question was straightforward:
Where would sufficient revenue come from to support the infrastructure being built?
Two years later, that question remains relevant, but its implications extend beyond infrastructure providers and frontier model developers.
It’s becoming a question for every company building AI-powered applications.
In January 2026, Epoch AI published an analysis examining whether AI companies could become profitable.
Using publicly reported information and financial assumptions, the researchers estimated that OpenAI’s offerings during the GPT-5 flagship period generated approximately $6 billion in revenue against $4 billion in inference compute expenditure.
That suggested a gross margin of roughly 30% after inference compute costs.
However, the researchers estimated that margins became substantially thinner after accounting for other operating expenditure, even before fully incorporating research and development costs.
These were independent estimates, not audited company financial results. The analysis also covered a bundle of OpenAI products, not GPT-5 inference alone.
Still, the broader lesson matters.
Generating revenue from AI inference is not the same as generating sustainable profits from AI products.
An AI business must support infrastructure, research, talent, distribution, reliability, and continued product development.
Increasing revenue is essential, but so is understanding what it costs to generate that revenue.
We Might Be Optimizing the Wrong Metrics
Product teams have become exceptionally good at evaluating AI models.
We compare accuracy, latency, reasoning capabilities, context windows, benchmark scores, and token prices.
These measurements are valuable.
But they don’t necessarily tell us whether an AI-powered workflow makes economic sense.
Consider two hypothetical AI systems performing the same task.
System A uses a relatively inexpensive model but requires multiple reasoning attempts, repeated retrieval operations, and additional validation steps.
System B uses a more expensive model but completes the task reliably with fewer executions.
Looking exclusively at token pricing might suggest that System A is more economical.
Looking at the total cost of completing the task could lead to a different conclusion.
This is supported by a 2025 research paper titled Economic Evaluation of LLMs, which argues that model selection should consider the economic consequences of errors, latency, and abstention alongside inference expenditure.
The researchers demonstrated that more capable models can become economically preferable when mistakes carry meaningful costs.
The findings were based on mathematical reasoning benchmarks, but the framework offers a useful way to think about enterprise AI.
The cheapest inference isn’t necessarily the cheapest successful outcome.
I believe this should fundamentally change how we evaluate AI product performance.
Instead of measuring only cost per token or cost per request, product teams should increasingly measure:
- Cost per completed task: Total computation and execution expenditure associated with delivering an outcome.
- Cost per successful task: Total expenditure, including retries and failures, relative to successfully completed tasks.
- Value generated per task: The measurable business benefit created by the completed workflow.
- Cost of errors: The operational, financial, or regulatory consequences of incorrect outputs.
- Contribution margin per workflow: Revenue attributable to the workflow minus the variable costs required to deliver it.
The objective is not simply reducing inference expenditure.
It’s maximizing the value generated by reliable AI execution.
Healthcare AI Makes This Problem Particularly Interesting
Consider a clinical intelligence platform.
A physician or care management professional asks:
“Which patients have potential documentation gaps, and what clinical evidence supports those findings?”
It appears to be a straightforward question.
But answering it accurately could require retrieving longitudinal patient records, examining clinical documentation, identifying relevant diagnoses, evaluating evidence, applying coding rules, and generating an explainable response.
Now imagine executing similar workflows across tens of thousands of patients.
What initially appears to be a conversational interface becomes a computationally intensive clinical intelligence system.
But here’s the important distinction.
Not every task in that workflow requires the same degree of AI reasoning.
Retrieving a patient’s most recent encounter might require a structured database query.
Identifying the latest documented HbA1c result could involve deterministic retrieval over normalized clinical information.
Evaluating whether a diagnosis has adequate supporting documentation might require semantic interpretation and more sophisticated reasoning.
Determining whether an HCC is valid under the applicable coding model also requires controlled reference logic, clinical evidence, and appropriate review.
Using the same expensive frontier model for every step might be unnecessarily costly.
However, using the cheapest possible model for every step would introduce a different problem.
In healthcare, incorrect outputs can have consequences beyond software performance.
An unsupported diagnosis could create compliance concerns.
An incorrectly interpreted clinical record could generate misleading recommendations.
An inaccurate quality measure assessment could affect reporting or operational decisions.
This means healthcare AI cannot optimize inference expenditure independently of accuracy, clinical safety, traceability, and regulatory requirements.
The economic objective becomes more specific:
Deliver the lowest sustainable cost per clinically accurate, evidence-grounded outcome.
That’s a substantially different problem from simply choosing the cheapest language model.
From Model Selection to Intelligent Orchestration
This brings us to what I believe will become an increasingly important architectural capability: intelligent orchestration.
Today, many AI systems are designed around a relatively straightforward pattern.
A user submits a request, relevant context is retrieved, and a language model generates an answer.
More sophisticated systems introduce agents, tool calling, retrieval-augmented generation, and multi-step reasoning.
But as these systems become more complex, we need to ask whether every request should follow the same computational pathway.
A more economically efficient architecture would first understand the nature of the request.
Can the answer be retrieved from validated structured data?
Can a deterministic rule produce the required result?
Does an existing, sufficiently current computation already contain the answer?
Would a smaller specialized model be sufficient?
Or does the request genuinely require more expensive frontier-model reasoning?
The answers to these questions should influence how the system executes the workflow.
This is not merely theoretical.
Research presented at ICLR 2025, RouteLLM: Learning to Route LLMs from Preference Data, demonstrated that intelligent routing between stronger and weaker models could reduce inference expenditure by more than twofold in certain benchmark settings while preserving measured response quality.
These savings are workload-dependent, and equivalent results should not be assumed in regulated clinical applications without validation.
Nevertheless, the research supports a broader architectural direction.
Rather than treating the language model as the default solution for every computational problem, AI systems can selectively allocate intelligence where it creates the most value.
Structured retrieval, caching, deterministic execution, specialized models, and frontier reasoning each have a role.
The challenge is orchestrating them intelligently while maintaining reliability.
And this has implications beyond engineering.
It affects product pricing, scalability, reliability, customer experience, and ultimately gross margins.
Model orchestration is becoming a product strategy decision, not simply an infrastructure optimization.
The Pricing Problem Nobody Can Ignore
There’s another dimension worth discussing.
AI agents are changing how software is consumed.
Traditional SaaS pricing commonly revolves around seats, subscriptions, usage tiers, or access to functionality.
But autonomous AI introduces a more complicated relationship between consumption and cost.
Two customers might purchase identical subscriptions while generating dramatically different computational workloads.
A user who executes a few simple tasks may be inexpensive to serve.
Another user who delegates complex, long-running workflows may consume significantly more resources.
If their revenue contribution remains similar, their margins can diverge.
This doesn’t necessarily mean subscription pricing is obsolete.
It means AI product companies need better mechanisms for connecting pricing, usage, and delivered value.
We may increasingly see hybrid approaches combining subscriptions, usage allowances, consumption-based billing, and outcome-linked pricing.
However, outcome-based pricing introduces its own challenge.
If the computational cost of completing a task is unpredictable, pricing that task profitably becomes difficult.
This is precisely why understanding workflow-level economics matters.
Before deciding how customers should pay, companies need to understand what delivering a successful outcome actually costs.
The Next Competitive Advantage
I don’t believe the conclusion from this research is that AI is inherently unprofitable.
Inference efficiency is improving rapidly.
Hardware continues to advance.
Models are becoming more capable.
Competition is creating opportunities to deliver comparable performance at lower prices.
And AI can generate substantial economic value by reducing operational effort, accelerating decisions, and enabling work that previously required significant human involvement.
The opportunity remains enormous.
But the economics cannot be an afterthought.
For AI product leaders, the challenge is evolving from demonstrating what an AI system can do to demonstrating what it can do reliably, repeatedly, and sustainably.
That requires product teams to understand not only model performance but also workload behavior, cost variability, customer value, and the economic consequences of errors.
It requires engineering teams to build systems that use computation deliberately rather than indiscriminately.
And it requires leadership teams to recognize that rapid adoption, impressive demonstrations, and increasing inference consumption are not necessarily indicators of sustainable business performance.
Elon Musk’s statement about Grok Bot may have been about a particular product and its serving costs.
But the underlying problem is much larger.
As AI becomes capable of performing increasingly complex work, the cost of delivering that intelligence becomes an integral part of the product itself.
The next competitive advantage in AI may not belong exclusively to the company with the smartest model.
It may belong to the company that delivers the most reliable intelligence, creates the greatest measurable value, and does so with sustainable economics.
And perhaps that’s the question AI product teams should be asking more often:
Are we building products that maximize intelligence, or products that maximize the economic value of intelligence?
Research and References
- Stanford HAI (2025) — Artificial Intelligence Index Report 2025. Research on declining inference prices and AI development economics.
- Stanford Digital Economy Lab (2026) — How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. Working paper on computational consumption across agentic software engineering workloads.
- Sequoia Capital (2024) — David Cahn, AI’s $600B Question. Analysis of infrastructure spending and the revenue required to support the AI ecosystem.
- Epoch AI (2026) — Can AI Companies Become Profitable? Lessons from GPT-5’s Economics. Independent estimates of model-serving revenue, inference expenditure, operating economics, and development costs.
- Zellinger & Thomson (2025) — Economic Evaluation of LLMs. Research on evaluating AI systems using the financial consequences of inference, errors, latency, and abstention.
- Ong et al. (ICLR 2025) — RouteLLM: Learning to Route LLMs from Preference Data. Research on optimizing inference expenditure through intelligent model routing.
- TechCrunch (November 2, 2026) — Reporting on Elon Musk’s comments about Grok Bot’s serving costs and usage limitations.
#ArtificialIntelligence #AgenticAI #AIProducts #ProductStrategy #AIEconomics #HealthcareAI
Leave a Reply