Skip to main content

Strictly necessary · always on

Remembers this choice in your browser. No tracking.

blog_post_13.10.png
AI Costs

Four of the Five AI Cost Levers Touch a Compliance Control

Marko Jagodic··Updated ·10 min read

Token prices fall while AI bills climb, and the standard cost-cutting playbook has five levers. In regulated work, four of them break a control or reopen a validation. Here's what that leaves you, and the number to manage instead of price per token.

Token prices keep falling while AI bills keep climbing, and most cost-cutting playbooks reach for the same five levers. In regulated work, four of them either break a control or reopen a validation, which leaves exactly one you can pull without a compliance conversation. This post walks through all five.

Back in June, TechCrunch's Rebecca Bellan collected the sentence that still describes the quarter. It's from Vitaly Gordon, CEO of Faros AI: "One of my engineers spent $40,000 on tokens last month, and I genuinely don't know whether I should stop him or should I go and tell everyone else to be like him."

That's not really a complaint about cost. It's a question about who in the building is allowed to say no, and nobody could answer it. J.R. Storment of the FinOps Foundation was hearing the same thing that spring from the buyer's side: "In April and May, I started hearing from companies: 'Oh my god, we are 3x over our entire 2026 token budget and it's only April.'" Uber, per the same piece, had blown its full 2026 AI coding budget before May was out.

For teams deploying AI into a hospital, a claims workflow, a vehicle, or a lending decision, the follow-up order has been arriving all year: cut the AI bill by a third. The trouble is that the fastest levers to pull are the ones holding up the compliance evidence.

Why the bill climbed while prices fell

The obvious explanation is model pricing, and it's the wrong one. Prices keep falling; the cost lives in the shape of the work. Stanford's Digital Economy Lab measured this in May (Longju Bai, Brynjolfsson, Pentland and Pei): agentic tasks burn roughly 1,000 times the tokens of a code-chat exchange, and running the same agent on the same task swings by up to 30x depending on how it happens to reason that day.

The spend also concentrates in input rather than output, because an agent re-reads the prompt and every prior response before each step, so context compounds and gets paid for again on every turn. Pei's conclusion is the uncomfortable one: "Agents are not capable of predicting their own token costs. This is the fundamental bottleneck for result-based pricing for agents."

Janakiram MSV made the accounting explicit in Forbes this July. Budgeting arguments collapse, he wrote, because they treat four separate quantities as one: price per token, tokens per attempt, total organizational spend, and cost per successful task, which divides the whole workflow's cost by the tasks that finished correctly and is the number that actually matters. His example is worth keeping: a pricier model that resolves 90% of tickets on the first try can cost less per resolved ticket than a cheap one that resolves 40%, retries, escalates, and lands on a human anyway.

So the rate card falls while the invoice climbs. The mismatch is a measurement error: the token is the unit being budgeted, and it isn't the thing being bought.

AI spend now goes through the CFO

Forrester's 2026 predictions called it early: enterprises are deferring 25% of planned AI spend into 2027, and CEOs are leaning on CFOs to approve what's left. Only 15% of AI decision-makers reported an EBITDA lift in the past twelve months, and AI already accounts for 18% of cloud spend at AI-forward enterprises, up from 4% in 2023.

Sastry Durvasula, Chief Operating, Information and Digital Officer at TIAA, put the ask plainly in CIO this summer: "You need to understand the full cost of operations, the efficiencies of running tokens or how you're handling traffic or RAG."

That's a reasonable ask. In regulated work, though, the full cost of operations includes a category that rarely made it into the pilot budget, and the honest way to see it is lever by lever.

The five levers, and what each one costs in regulated work

Every cost-reduction playbook converges on the same five moves. Four of them run straight into a control.

SPFR_CostLevers_table.png

Switching to a cheaper model sounds free until you remember that in a cleared device the model is the subject of the submission, not a dependency to swap. FDA made this point twice within about ten weeks this year. In April it denied a petition to exempt radiology AI from 510(k) review, reasoning that clearing one model doesn't predict the safety of the next. Then on June 17 it published a final order putting a Predetermined Change Control Plan inside the device type itself: radiological ML quantitative imaging software with a PCCP is now its own Class II classification. The order is enabling, since it's the sanctioned path for changing a model without a new submission, but it also names the price of admission: model swaps happen along a route that was pre-declared and validated. The token savings are real, and the re-validation is a cost that never appears on the rate card. We've written before about teams that end up freezing the model rather than paying it.

Cutting the human review step is the biggest line on most budgets, and legislation is steadily removing it as an option. Indiana's HB 1271 took effect July 1: insurers can't use AI as the sole basis to downcode a claim without a human reviewing the medical record, and providers can't submit AI-generated claims without human review either, so the requirement cuts both ways and is already in force. Utah's SB 319 is the one to plan for: signed in March and effective January 1, 2027, it requires adverse preauthorization determinations to come from someone exercising independent medical judgment who "does not rely solely on recommendations from any other source." Those reviewer minutes aren't overhead on the product. In this workflow they are the product.

Sampling or shortening the logs looks like pure overhead until something fails in production and someone has to reconstruct, months later, which model version saw which input and produced which output for one specific decision. Sampled logs can't answer that, and the clock on that reconstruction starts immediately. In a regulated product the observability stack is evidence, not telemetry, and evidence thinned to save on storage is evidence you don't have.

Caching aggressively changes what a response is. A cached inference is a decision made on stale inputs, and in a clinical or credit workflow that's a different decision from the one that was validated, served fast enough to look like the same thing.

Trimming context and retries is the one lever that's usually safe, and it has real money attached, since input tokens dominate the bill. The caveat is that trimming the wrong piece of context changes the model's behavior, which reopens the validation loop the trim was supposed to help pay for.

Four of the five either break a control or reopen a validation, which is close to a definition of what makes a workflow regulated in the first place: the expensive parts are load-bearing.

Cheaper tokens, worse ratio

This part rarely makes it into a deck, probably because it's arithmetic rather than a story. A regulated AI workflow has a unit cost with two halves. The variable half (tokens, inference, retries) is deflating fast. The fixed half (reviewer minutes, the eval suite, retained logs, traceability, the re-validation triggered by every change) doesn't move when a model gets cheaper, because a clinician still reads at the speed of a clinician.

Every vendor price cut therefore makes compliance a larger share of cost per decision, not a smaller one. The cheaper the model gets, the more of the AI bill is simply the price of proving the thing works.

One caution before budgeting around this with a hoped-for clearance date. The Congressional Research Service wrote on June 10: "To date, the agency does not appear to have authorized for marketing any generative AI (genAI)-enabled device." Roughly 1,450 AI-enabled devices carry FDA authorization; on CRS's reading, none of them are generative. FDA granted a patient-facing clinical genAI app breakthrough designation back in March, but designation is not authorization. A roadmap that treats genAI clinical clearance as a schedulable milestone with a known price tag is pricing against a precedent that doesn't exist yet.

What you can actually cut

Plenty, once the controls come off the table.

Most regulated AI workflows send every step to a frontier model, including steps that make no decision at all: parsing, routing, extraction, formatting. None of those are regulated acts. They can run on small models, or on plain code, which frees up budget without touching anything an auditor cares about. The decision step carries the evidence burden, and it's usually a fraction of total traffic.

Then attack attempts-per-task instead of price-per-token. This is what the eval harness is actually for: an agent that succeeds on the first try costs less than a cheap agent that needs four attempts to get there. In regulated work that eval suite was getting built anyway, so the compliance evidence and the cost-control instrument turn out to be the same file.

Scope is the third cut. A narrow agent with a deterministic pre-filter beats a broad agent with a huge context window on both cost and defensibility. Most AI business cases get built backwards, and this is what backwards costs in dollars.

Computing the real cost per task

  1. Compute one number: cost per successful, defensible task, not cost per token. Divide the whole workflow's cost (inference, retries, review minutes, log storage) by the tasks that finished correctly and left evidence behind. It's the only number a CFO can act on.
  2. Split the bill into a deflating column and a fixed column. Tokens go on the left; reviewer time, retention, eval runs and re-validation go on the right. If the right column isn't in the budget at all, that's the finding.
  3. Find the steps that make no decision. Walk one workflow end to end and mark every model call that isn't a regulated act. Those can be optimized today, without anyone's sign-off.
  4. Price the next model swap before wanting one. What would re-validation cost, in weeks and in dollars? A team that can't answer that doesn't have a PCCP strategy; it has a model it's afraid to touch.
  5. Before cutting retention, run the regulator test. Pick a real decision from ninety days ago and try to reconstruct its version, input and output. If that fails, sampling the logs was never a saving, just deferred liability.

The 85% is also the bill

This series has spent months arguing that the model is maybe 15% of the work and the other 85% (integration, monitoring, evidence) is what actually ships. That argument is close to consensus now. The less popular half is that the 85% is also most of the bill, and it's the part you're not allowed to cut. Nobody designed it that way; it's simply what you're buying when you deploy AI somewhere failure has consequences.

That makes this a design problem rather than a procurement one. The teams whose unit economics are under control got there upstream: narrow scope, the model kept out of steps that don't need it, one eval harness doing double duty as evidence and cost control. The teams still rationing tokens this fall priced the model instead of the workflow.

We've built that discipline into FDA-regulated device data before, on the recovery-intelligence platform we shipped with Canary Medical, where the eval harness doubles as the audit trail. A First Build is about a month against your real data, at a fixed price. You leave with a working milestone, an eval harness that proves it works, and an evidence plan with the regulatory path built in. The harness is the part that matters here: it's the evidence a reviewer will ask for and the instrument that tells you what a successful task actually costs, one file doing both jobs. If your next budget conversation is about to cut something load-bearing, scope a First Build before the cut lands somewhere it shouldn't.

Tagged:AI CostsFinOpsRegulated IndustriesAI GovernanceHealthcare
Explore deployments in this vertical:Healthcare reference architecture
Your Turn

Facing a problem like this one?

Tell us what the business needs to win and who owns it on your side. If it is a fit, a First Build starts within weeks. If it is not, we will tell you why.