**Four of the six options surface token usage automatically**, meaning you can subscribe to an event or stream part and get usage data without reading the raw response object.
- **LangChain** exposes usage through `with_usage_metadata()`, the LangSmith trace surface, or callback events (`on_llm_end` carries usage metadata). Wire a callback once, get usage on every call.
- **Vercel AI SDK** puts usage on the `finish` stream part for `streamText` and on the return object for `generateText`. You consume it as part of the normal API.
- **LlamaIndex** provides `TokenCountingHandler`, a callback that aggregates usage across the QueryEngine pipeline. Drop it in once, query the totals later.
- **Mastra** emits OpenTelemetry GenAI spans that include token usage by default. Wire OTel to Langfuse or Phoenix and usage shows up alongside traces.
The two that do not auto-surface:
- **Raw OpenAI SDK** puts usage on `response.usage`. You can read it, but the SDK does not log or aggregate it for you. That is on you.
- **DSPy** tracks usage during `compile()` for cost-aware optimization, but at runtime you instrument your own usage capture.
The trap to avoid: even the four 'automatic' frameworks only solve capture, not aggregation. You still need to roll it up by request, by user, by feature, by day.
**Four frameworks surface usage automatically, two do not.**
**LangChain** exposes token usage through three channels: the `usage_metadata` field on `AIMessage` (post-v0.3, available via `with_usage_metadata()` on streaming chains), callback events (`on_llm_end` carries usage in the response), and LangSmith traces (each span has usage broken out). Wire a callback handler once and every chain emits usage.
**Vercel AI SDK** surfaces usage on the `finish` part of `streamText` streams and on the return object of `generateText`. The shape includes `promptTokens`, `completionTokens`, `totalTokens`, and per-tool usage when tools ran. The `experimental_telemetry` option emits OpenTelemetry spans with the same data for backend dashboards.
**LlamaIndex** ships `TokenCountingHandler` as a CallbackManager handler. Drop it into the QueryEngine and it aggregates per-call usage across retriever, postprocessor, and synthesizer invocations.
**Mastra** is OTel-native. Every agent call emits a GenAI semantic-convention span with `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Pipe OTel to Langfuse, Phoenix, or Honeycomb and usage is in your dashboards without extra wiring.
**Raw OpenAI SDK** puts usage on `response.usage` for chat completions and on stream chunks (last chunk for SSE). The SDK does not log it. You have to read it, structure it, and ship it to your metric store. Exposing is not surfacing.
**DSPy** uses usage during `compile()` for cost-aware optimizers (BootstrapFewShot tracks per-call tokens). At runtime, the compiled program calls the underlying LM (OpenAI, Anthropic) and you instrument that LM client yourself if you want runtime usage.
The production insight: even auto-surfacing frameworks only solve capture. Aggregation (per-user, per-feature, per-day), attribution, and dashboarding are still your responsibility. Typically by routing usage through OpenTelemetry to a backend like Langfuse, Phoenix, or Helicone.
Quick reference
TL;DR
LangChain, Vercel AI SDK, LlamaIndex, and Mastra emit token-usage events you subscribe to; raw OpenAI and DSPy expose usage on the response or compile loop but leave runtime aggregation to you.
Think of receipts at a diner. Some restaurants email you the receipt automatically when you leave (LangChain, Vercel AI SDK, LlamaIndex, Mastra). Others print it and leave it on the table. You can read it and file it, but if you forget, no record exists (raw OpenAI SDK). DSPy is a restaurant that tracks your bill carefully during a tasting menu (the compile loop) but leaves you to track everyday meals on your own. All of them know the price; only some hand it to you without asking.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Backend service where every call already goes through your own logging middleware Framework auto-capture duplicates work you have done, you may prefer to read the SDK's response.usage directly to keep one source of truth for cost data.
Multi-tenant proxy architecture where users bring their own model keys Capture at the proxy (Helicone, OpenRouter) is cleaner than per-framework callbacks because it sees every call regardless of how the client is structured.
Highly cost-sensitive batch workloads where compile-time optimization is the goal DSPy's compile-time usage tracking is exactly the right tool, runtime auto-capture frameworks add overhead without comparable cost-reduction signal.
Quick scan · 2–4 min
Everything important, quickly.
6 to 8 min: four auto-surfacing frameworks + two non-auto + streaming caveat + capture vs aggregation distinction + 2026 OTel-routed production pattern.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith and Langfuse render LangChain usage_metadata as cost per trace without extra wiring once you set the LANGCHAIN_TRACING_V2 env var.
2Vercel AI SDK ships experimental_telemetry that emits OTel GenAI spans, Vercel's own demos pipe these into a Langfuse dashboard.
3LlamaIndex's TokenCountingHandler is the canonical RAG cost-tracking pattern in production code, attached at the QueryEngine level.
4Mastra's quickstart instruments OTel from minute one, the dashboards show usage broken down by agent and tool out of the box.
5Helicone proxies raw OpenAI SDK calls and captures usage at the proxy layer for teams that do not want to instrument the SDK themselves.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you aggregate token usage per user per feature across a stack that uses both LangChain and Vercel AI SDK?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the cleanest way to track token usage for a streaming Vercel AI SDK chat with multiple tool calls?
Sign in or go Pro to see the approach for this follow-up.
QIf you wanted to add runtime usage capture to a DSPy-compiled program, how would you do it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the framework solves cost reporting end to end. It captures usage; you still own aggregation, attribution, and the dashboard.