Monitoring token usage and cost across a multi-step AI workflow comes down to three practices: record usage at every model call and tool step, attach consistent identifiers so those records can be aggregated into a trace or thread, and treat locally computed cost figures as estimates that must be reconciled against provider billing. A single prompt-and-response is easy to measure. A workflow that retrieves documents, plans, calls tools, generates an answer, and then runs an evaluator is not, because spend is spread across steps that may use different models, different token types, and non-LLM components.
This guide is written for practitioners who already have a working multi-step pipeline — an agent, a RAG chain, or an automation sequence — and want per-step visibility instead of a single number at the end. The mechanics described here follow documented behavior in LangSmith’s cost tracking and the Claude Agent SDK’s usage reporting. The workflow walkthrough is an illustrative design, not a report of a specific deployment.
Why an aggregate total is not enough
A total tells you that you spent money. It does not tell you which step spent it, which is the question that actually drives optimization. Multi-step workflows make this worse in two ways. First, cost concentrates unpredictably: a planning step may be cheap while a retrieval-augmented generation step with a large context window dominates the bill. Second, some steps do not consume LLM tokens at all but are still billable — a web search request, a vector database query, or a third-party API call.
LangSmith’s cost model addresses this by splitting usage into three categories. Input covers tokens in the prompt sent to the model, with subtypes including cache reads, text tokens, and image tokens. Output covers tokens generated in the response, with subtypes including reasoning tokens, text tokens, and image tokens. Other covers costs from tool calls, retrieval steps, or any custom runs. That third bucket is the one most multi-step pipelines forget to populate, which is why their cost dashboards under-report.
What to capture at each step
Before wiring anything up, decide on a consistent record shape. A practical minimum for each step in the workflow:
- A step or run identifier and its type (model call, tool call, retrieval, evaluator).
- The model or provider identifier, including version where the provider exposes it.
- Input tokens, broken out by subtype where available, including cache reads.
- Output tokens, broken out by subtype, including reasoning tokens.
- Any per-request fee that is not token-based, such as a web search charge.
- Custom cost values for non-LLM components you are tracking yourself.
- Parent and root identifiers, plus a thread or session identifier.
- A flag recording whether the cost figure is a client-side estimate or provider-reported.
That last item is not bookkeeping pedantry. As described below, the two numbers can legitimately differ.
Instrumenting the workflow, step by step
1. Define step boundaries before adding code
Each model call and each tool or retrieval call should become its own recorded run. LangSmith’s trace tree is built on this assumption: it displays total usage for the entire trace, aggregated values for each parent run, and token and cost breakdowns for each child run. If your instrumentation only wraps the top-level function, you get the aggregate and lose the breakdown.
2. Capture usage from every model response
Most provider SDKs return usage on the response object. The Claude Agent SDK exposes per-step token breakdowns on each assistant message — message.message.id and message.message.usage in TypeScript, message.message_id and message.usage in Python — plus per-model cost via modelUsage (TypeScript) or model_usage (Python) on the result message, and a cumulative total on the result message. Persist the per-step values, not just the cumulative one; the cumulative figure cannot be decomposed after the fact.
3. Attach thread metadata to child runs
This is the single most common reason thread-level cost reports look wrong. LangSmith’s documentation is explicit: when tracking costs across threads, all child runs must include the thread metadata (session_id or thread_id). Without thread metadata on child runs, token counts and costs from those runs are not included in thread-level aggregations. A workflow whose root run carries the thread ID but whose tool calls do not will silently under-report.
4. Submit custom costs for non-LLM components
LangSmith permits submitting custom cost data for components beyond LLM calls, which is how retrieval steps, external APIs, and tool invocations get into the same unified view. If a component has a fixed per-call price or a metered rate you can compute, record it as a custom cost rather than leaving it at zero.
5. Configure model pricing
Cost figures are only as good as the price table behind them. In LangSmith, evaluator spend tracking and limits use the per-model pricing configured under Model pricing. The Claude Agent SDK computes cost locally from a price table bundled at build time, unless a modelPricing table is in effect. Any model the price table does not recognize will produce a wrong or missing number.
6. Aggregate and review at the right level
Use three views: the trace tree for a single execution, project-level stats for aggregate trends, and dashboards for longer-horizon monitoring. Reviewing only the project total hides which step regressed after a prompt or retrieval change.
An illustrative multi-step workflow
Consider a five-stage assistant: (1) retrieve passages from a vector store, (2) plan the answer with a small model, (3) call an external search tool, (4) generate the final answer with a larger model, and (5) run an evaluator that scores the output. Stages 2 and 5 consume tokens; stage 1 consumes no LLM tokens but has its own cost; stage 3 may incur a per-request fee. A per-step record for stage 4 might look like this:
{
"run_id": "gen-8f21",
"step": "generate_answer",
"model": "provider/model-name",
"input_tokens": 1840,
"input_cache_read_tokens": 1500,
"output_tokens": 320,
"output_reasoning_tokens": 0,
"other_cost_usd": 0.0,
"thread_id": "thread-42",
"parent_run_id": "trace-root"
}The point of the example is the shape, not the values: cache-read tokens are separated from fresh input tokens, non-token costs have their own field, and the thread identifier is present so the record can be rolled up. With records like this at every stage, a cost increase can be attributed to the stage that caused it — for instance, a jump in fresh input tokens after a retrieval change increased the context size.
Estimates versus authoritative billing
Client-side cost numbers are useful for development and budgeting, but they are not invoices. The Claude Agent SDK documentation states that total_cost_usd and costUSD are client-side estimates, not authoritative billing data, computed locally from a price table bundled at build time unless a modelPricing table is in effect. The documented causes of drift are:
- Pricing changes after the SDK was built.
- The installed SDK version does not recognize a model.
- Billing rules that the client cannot model.
One billing rule the SDK does model is data residency pricing: when a response’s usage reports inference_geo: "us", the SDK multiplies the list price of that response’s tokens by 1.1. Per-request fees such as web search are not multiplied. This behavior requires TypeScript Agent SDK v0.3.239 or later, or Python Agent SDK v0.2.144 or later. The same documentation advises using the Usage and Cost API or the Usage page in the Claude Console for authoritative billing, and warns against billing end users or triggering financial decisions from the estimate fields.
The practical implication for multi-step workflows is that your monitoring dashboard and your invoice answer different questions. The dashboard answers “which step changed?” The invoice answers “what do I owe?” Reconcile them periodically rather than assuming they match.
Guardrails, limits, and troubleshooting
Once per-step data exists, guardrails become possible. LangSmith supports tracking and limiting evaluator spend, using the per-model pricing configured under Model pricing — useful when an evaluator runs on every step of a long workflow. Beyond platform-level limits, common practice is to set alert thresholds on thread-level cost so a runaway loop is caught before it becomes expensive.
When the numbers look wrong, work through these checks in order:
- Thread totals lower than the sum of visible steps. Check for child runs missing
session_idorthread_id; they are excluded from thread-level aggregation. - Cost is zero or implausible for a specific model. The local price table likely does not recognize the model. Supply a
modelPricingtable or upgrade the SDK. - Cost per step looks inflated after enabling prompt caching. Cache reads are a distinct input subtype; confirm they are being recorded and priced separately from fresh input tokens.
- Parallel tool calls produce confusing counts. Attribute usage per step rather than per conversation turn, which is what the per-step breakdowns on assistant messages are for.
- Local estimate diverges from the invoice. Check data residency multipliers, per-request fees that are not token-based, and any pricing change newer than the installed SDK.
Where to start
Instrument one stage first — usually the generation step, because it tends to dominate cost — and verify its recorded token counts against the provider’s own usage reporting before expanding. Then add thread metadata to every child run, populate the “other” cost bucket for retrieval and tool steps, and only then build dashboards. A workflow with accurate per-step records can answer cost questions in minutes; one with only a total cannot answer them at all.
If you are working through this on your own stack, the community forum is a good place to share your step breakdown, the platform or SDK you are using, and where the recorded numbers diverge from what you expected.
FAQ
Why is my thread-level cost lower than the sum of the steps I can see?
The most likely cause is missing thread metadata. LangSmith documents that child runs must include session_id or thread_id; without it, their token counts and costs are excluded from thread-level aggregations.
Can I bill users from the cost fields the SDK returns?
No. The Claude Agent SDK documentation describes total_cost_usd and costUSD as client-side estimates and advises against billing end users or triggering financial decisions from them. Use the provider's Usage and Cost API or console usage page for authoritative figures.
Why does my recorded cost differ from the invoice even when token counts match?
Documented causes include pricing changes newer than the installed SDK, a model the bundled price table does not recognize, and billing rules the client cannot model. Data residency pricing is one rule the SDK does model: usage reporting inference_geo: "us" multiplies that response's token list price by 1.1, while per-request fees such as web search are not multiplied.
Does prompt caching change how I should read cost reports?
It changes the composition of input cost rather than removing it. LangSmith treats cache reads as a subtype of input tokens, so separating cache-read tokens from fresh input tokens is what lets you tell whether a caching change actually reduced spend.