Identify the LLM frameworks that surface token usage automatically (no manual wiring needed), select all that apply

Multi-select·Medium·4.0 · 0·~1 min·Asked atHebbia Humanloop Roblox

Attempt it

**Four of the six options surface token usage automatically**, meaning you can subscribe to an event or stream part and get usage data without reading the raw response object. - **LangChain** exposes usage through `with_usage_metadata()`, the LangSmith trace surface, or callback events (`on_llm_end` carries usage metadata). Wire a callback once, get usage on every call. - **Vercel AI SDK** puts usage on the `finish` stream part for `streamText` and on the return object for `generateText`. You consume it as part of the normal API. - **LlamaIndex** provides `TokenCountingHandler`, a callback that aggregates usage across the QueryEngine pipeline. Drop it in once, query the totals later. - **Mastra** emits OpenTelemetry GenAI spans that include token usage by default. Wire OTel to Langfuse or Phoenix and usage shows up alongside traces. The two that do not auto-surface: - **Raw OpenAI SDK** puts usage on `response.usage`. You can read it, but the SDK does not log or aggregate it for you. That is on you. - **DSPy** tracks usage during `compile()` for cost-aware optimization, but at runtime you instrument your own usage capture. The trap to avoid: even the four 'automatic' frameworks only solve capture, not aggregation. You still need to roll it up by request, by user, by feature, by day.

**Four frameworks surface usage automatically, two do not.** **LangChain** exposes token usage through three channels: the `usage_metadata` field on `AIMessage` (post-v0.3, available via `with_usage_metadata()` on streaming chains), callback events (`on_llm_end` carries usage in the response), and LangSmith traces (each span has usage broken out). Wire a callback handler once and every chain emits usage. **Vercel AI SDK** surfaces usage on the `finish` part of `streamText` streams and on the return object of `generateText`. The shape includes `promptTokens`, `completionTokens`, `totalTokens`, and per-tool usage when tools ran. The `experimental_telemetry` option emits OpenTelemetry spans with the same data for backend dashboards. **LlamaIndex** ships `TokenCountingHandler` as a CallbackManager handler. Drop it into the QueryEngine and it aggregates per-call usage across retriever, postprocessor, and synthesizer invocations. **Mastra** is OTel-native. Every agent call emits a GenAI semantic-convention span with `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Pipe OTel to Langfuse, Phoenix, or Honeycomb and usage is in your dashboards without extra wiring. **Raw OpenAI SDK** puts usage on `response.usage` for chat completions and on stream chunks (last chunk for SSE). The SDK does not log it. You have to read it, structure it, and ship it to your metric store. Exposing is not surfacing. **DSPy** uses usage during `compile()` for cost-aware optimizers (BootstrapFewShot tracks per-call tokens). At runtime, the compiled program calls the underlying LM (OpenAI, Anthropic) and you instrument that LM client yourself if you want runtime usage. The production insight: even auto-surfacing frameworks only solve capture. Aggregation (per-user, per-feature, per-day), attribution, and dashboarding are still your responsibility. Typically by routing usage through OpenTelemetry to a backend like Langfuse, Phoenix, or Helicone.

TL;DR

LangChain, Vercel AI SDK, LlamaIndex, and Mastra emit token-usage events you subscribe to; raw OpenAI and DSPy expose usage on the response or compile loop but leave runtime aggregation to you.

Memory aid

Easy to grasp

Think of receipts at a diner. Some restaurants email you the receipt automatically when you leave (LangChain, Vercel AI SDK, LlamaIndex, Mastra). Others print it and leave it on the table. You can read it and file it, but if you forget, no record exists (raw OpenAI SDK). DSPy is a restaurant that tracks your bill carefully during a tasting menu (the compile loop) but leaves you to track everyday meals on your own. All of them know the price; only some hand it to you without asking.

Key concepts

LLM Frameworks

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cost tracking is one of the most under-modeled production concerns in LLM stacks, and the framework taxonomy directly affects how much glue code you write. The multi-select tests whether you can distinguish 'the SDK exposes usage on the response' from 'the framework emits a usage event you subscribe to'. A small distinction that scales into either zero or hundreds of lines of instrumentation.

The answer is four out of six. Identifying which four is the entry-level test; articulating the capture vs aggregation distinction and the OpenTelemetry production pattern is the senior signal.

Mental model: capture is the framework's job, aggregation and dashboarding are yours. Frameworks that auto-capture save instrumentation glue; they do not save you from owning the cost-tracking pipeline.

The four that auto-surface usage

LangChain

Three complementary channels:

AIMessage.usage_metadata on the response (post-v0.3). The cleanest path for non-streaming calls.
with_usage_metadata() for streaming chains, since token deltas are partial and the aggregate only lands at the end.
Callback handlers (the on_llm_end hook on BaseCallbackHandler) carry usage in the response. LangSmith and Langfuse subscribe to these to render per-trace cost.

Set LANGCHAIN_TRACING_V2=true and the LangSmith trace surface gives you per-step usage for every chain without writing any code.

Vercel AI SDK

The TypeScript-first answer. streamText emits a finish part with:

finish: {
  usage: { promptTokens, completionTokens, totalTokens },
  finishReason: 'stop' | 'tool_calls' | 'length' | ...,
  // toolUsage when tools ran
}

generateText returns the same shape synchronously. Setting experimental_telemetry: { isEnabled: true } emits OpenTelemetry GenAI spans for backend dashboards.

LlamaIndex

TokenCountingHandler is a CallbackManager handler that aggregates usage across the QueryEngine pipeline:

python

from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
token_counter = TokenCountingHandler()
Settings.callback_manager = CallbackManager([token_counter])
# ... run queries ...
print(token_counter.total_llm_token_count)
print(token_counter.total_embedding_token_count)

Less granular than LangChain's per-span breakdown, but the right shape for RAG pipelines where you want one aggregate for the full retrieve and synthesize cycle.

Mastra

Native OTel from the start. Every agent call emits a GenAI semantic-convention span:

code

gen_ai.system: openai
gen_ai.request.model: gpt-5.5
gen_ai.usage.input_tokens: 1234
gen_ai.usage.output_tokens: 567

Wire OTel to Langfuse, Phoenix, Honeycomb, or a Datadog APM and usage shows up alongside traces with zero per-call instrumentation.

The two that do not

Streaming caveats and per-framework gotchas

The 2026 production pattern

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangSmith and Langfuse render LangChain usage_metadata as cost per trace without extra wiring once you set the LANGCHAIN_TRACING_V2 env var.
Vercel AI SDK ships experimental_telemetry that emits OTel GenAI spans, Vercel's own demos pipe these into a Langfuse dashboard.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you aggregate token usage per user per feature across a stack that uses both LangChain and Vercel AI SDK?

Route both into a common OpenTelemetry pipeline. LangChain emits via the LangSmith or OTel exporter; Vercel AI SDK emits via experimental_telemetry. Tag every span with user_id and feature_id as span attributes. Aggregate in a backend (Langfuse, Phoenix, or a SQL query over a span-export store) by those attributes. The framework boundary disappears at the dashboard layer.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming the framework solves cost reporting end to end. It captures usage; you still own aggregation, attribution, and the dashboard.

60 second bullets to scan on the way to the call.

The four frameworks that auto-surface usage and how each one exposes it
Why raw OpenAI SDK is exposing but not surfacing

Primary sources. Browse if you want the original framing.