Insights · Field Notes 04
Token cost: $/1M tokens? No!
Recently, my co-founder (ex-CFO, HBS) and I were talking with an FP&A team about how enterprises should measure AI costs.
The lazy answer is $/million tokens. That's what model providers (OpenAI, Anthropic, etc.) use, so teams naturally track it.
We believe different AI workflows need different KPIs:
- Automation: $/transaction
- Product feature (e.g., chatbot): $/active user
- Evals: $/eval
- Agent: $/completed job
Also, the metric you use to judge AI economics may not be the same one you use to control spend while AI is running.
Naturally, you need a control unit for the active workflow/job. But you also need a reporting unit for Finance to assess the economics after the work finishes.
- During the work: cost per active session/workflow
- After the work: cost per successful outcome (including retries, etc.)
Different teams will optimize for different KPIs locally. But ultimately, the single most important question engineers, ops, and finance teams need to answer is:
What did it cost to produce the outcome we wanted?
In Gateway or Not?, we go deeper into the technical details, the suboptimal approaches we see today, and what we believe is the right solution.
