Tracing AI API calls means recording a structured, queryable record of every model request your application makes — prompts, token counts, latency, tool calls, and metadata — organized into traces and spans rather than scattered plaintext log lines. That structure is what turns three vague complaints into answerable questions: why was this response slow, where did the tokens go, and what actually failed.
Whether you are calling a hosted model, running a local model behind an OpenAI-compatible endpoint, or wiring AI calls into a WordPress automation stack, the starting move is the same and deliberately small: instrument one high-traffic endpoint with an SDK wrapper or an OTLP exporter, then confirm the traces are landing somewhere you can query before expanding to full coverage [1].
What a trace records that a log line cannot
A trace is the whole request journey; a span is one timed step inside it. Each span records a parent, a start time, a duration, a status, and a controlled set of attributes [2]. That parent/child structure is what lets you see that a 12-second response was 9.5 seconds of model generation and 2 seconds of retrieval, instead of guessing.
Two identifiers matter, and they are not the same thing. Create a trace ID at the client or trusted edge, and a distinct request ID for each model attempt [2]. Retries and agent loops mean one user-facing task can contain several model calls, so a single ID per task would collapse exactly the detail you need. If an upstream provider returns its own identifier, store it as an external reference without assuming a particular field name [2].
Useful span attributes include application, environment, workload class, requested and actual model configuration, routing region, streaming mode, prompt version, tool-set version, release, and experiment cohort [2]. Those are dimensions you can group and compare by. Raw prompts, email addresses, and document names are not — high-cardinality and sensitive values belong in protected traces, if collected at all [2].
Capture these five fields before anything else
Choosing a trace schema with core fields — request ID, prompt, model, token counts, and latency — ensures the data you collect is meaningful and makes troubleshooting easier as coverage grows [1]. Before adding exotic attributes, confirm these five exist on every call, including the ones that fail.
Once they do, add a status field that is richer than a boolean. That single addition is what makes the failure section below possible.
Debugging latency: it is not one number
LLM API latency is not a single value. A user may wait too long for the first visible token, experience slow generation after it starts, or hit delay in retrieval, queueing, retries, validation, or frontend rendering [2]. A useful observability design connects traces, TTFT, TPOT, end-to-end latency, RPM, concurrency, token usage, errors, and cost, then compares distributions by workload, model, region, and version [2].
Splitting the number into TTFT and TPOT
For a streaming request, time to first token (TTFT) runs from the client beginning the request until it receives the first user-displayable content token [2]. Note where that window starts: at the client, not at the server, so connection setup and upload are part of the measurement [2]. A slow TTFT points at queueing, routing, retrieval, or prefill — not at generation speed.
TPOT, time per output token, describes what happens after the first token arrives. A request with fast TTFT and slow TPOT feels responsive at first and then drags; the fix usually lives in model size, quantization, batch contention, or concurrency limits rather than in your prompt.
Trace both separately. Averages hide the problem: a p50 that looks healthy can sit next to a p95 dominated by retries and long-context requests. Group by workload class and model version before concluding anything.
Debugging token usage and cost
Token counts belong on the span, split into prompt and completion tokens, and attached to the specific attempt rather than the task. That distinction matters because retries and multi-step agent workflows multiply calls: an agent trace may contain several model attempts, so call-level and task-level outcomes should be reported separately [2].
With per-attempt token counts, cost attribution becomes mechanical. Sum tokens by feature, prompt version, or experiment cohort and you can answer “which part of the product is spending the budget” without reading logs by hand. Linking evaluation scores directly to traces then ties quality regressions to specific prompts and model versions [1], so a cost change and a quality change can be reviewed together rather than in isolation.
Storage is the other half of the cost equation. Sampling policies and PII redaction are essential to control storage costs and protect sensitive information in large-scale tracing [1]. Sampling aggressively at the edge and keeping full detail only for errors or a defined slice of traffic is a common compromise.
Debugging failures: HTTP 200 is not task success
This is the most common blind spot in AI observability. A request can return a clean 200 and still have produced nothing usable. Distinguish completed, canceled, timed out, rate-limited, unauthorized, invalid, upstream failure, parse failure, schema failure, tool failure, and business rejection [2].
Those categories point to different owners and different fixes:
- Rate-limited or timed out — capacity, retry policy, or concurrency settings.
- Parse or schema failure — output formatting, validation, or a model that ignored the requested structure.
- Tool failure — an external dependency inside an agent loop, not the model itself.
- Business rejection — the system worked and the answer was still unacceptable to the product’s rules.
Record the status on the span, not just in an application log, so failure rates can be sliced by model version and prompt version alongside latency.
Choosing a capture method
There are three broad approaches: SDK wrappers, an OTLP exporter, and HTTP interception [1]. In practice the choice is about control and coverage. Wrappers are precise and easy to enrich with application context, but they require touching code around each client library. An OTLP exporter is vendor-neutral and fits naturally if you already run a collector or an observability backend across services. HTTP interception catches calls made by code you do not control, at the cost of seeing less application context.
A reasonable path is to start with a wrapper on one endpoint, confirm the trace schema, then standardize on OTLP once more than one service is involved.
Tracing local models and self-hosted runtimes
Local inference changes where the data lives, not what you should measure. If your runtime exposes an OpenAI-compatible API, the same wrapper pattern captures request ID, model, token counts, and latency, and the trace stays on your own infrastructure — which simplifies PII handling but leaves retention and storage entirely your responsibility. Local runtimes also expose their own logs and metrics; correlating those with your application trace ID is what connects “the model was slow” to “this specific user request was slow.”
Treat this as general practice rather than a tested configuration: exact metric names and endpoints differ by runtime and version, so verify against your own server’s documentation before relying on a specific field.
Trace IDs through WordPress and automation chains
Automation stacks are where traces most often break. A request may begin as an HTTP call, get queued by a scheduler such as WP-Cron, pass through a webhook, and only then reach the model. If the trace ID is not carried in the job payload, the model call becomes an orphan span with no user-facing context.
Two practical rules help. First, generate the trace ID at the trusted edge and propagate it through every hop, including queued jobs and retries. Second, keep post titles, file names, and user identifiers out of metric labels — they are high-cardinality and often sensitive — and store them only inside protected trace payloads if you need them at all [2].
An illustrative trace
The following is a constructed example, not a measured result. It shows the shape of a trace for a scheduled post-summarization job where the first model attempt was rate-limited:
trace_id: 7f3c1a... (illustrative)
span: job.run workload=wp-cron env=prod
span: retrieval.fetch_posts 412 ms status=ok
span: llm.call attempt=1 3180 ms status=rate_limited
span: llm.call attempt=2 9540 ms status=ok
ttft=1240 ms tpot=38 ms
prompt_tokens=3410 completion_tokens=218
span: validation.schema 6 ms status=okRead as a whole, this trace answers the three core questions at once: the task took roughly 13 seconds, most of it in a retried model call; the retry was caused by rate limiting rather than a bad prompt; and the token cost of the task includes both attempts, not just the successful one.
A short verification checklist
- Does every model call produce a span with a parent, start, duration, and status?
- Is there one trace ID per task and one request ID per attempt?
- Are prompt and completion tokens recorded per attempt?
- Is TTFT measured from the client, and is TPOT recorded separately?
- Can you filter failures by category, model version, and prompt version?
- Are sampling and redaction rules applied before storage?
If you can answer yes to the first three, you already have enough to debug most latency and cost complaints. The remaining items are what keep the system affordable and safe as traffic grows.
If you are working through this on your own stack, share your environment, the trace schema you settled on, and the failure category that surprised you most in the community discussion — concrete span examples from real setups are more useful to other builders than any generic checklist.
FAQ
Should I use one trace ID per task or one per API call?
Both, at different levels. Create a trace ID at the client or trusted edge for the whole task, and a distinct request ID for each model attempt, so retries remain visible as separate spans [2].
Can I log raw prompts as metric labels?
No. Raw prompts, email addresses, and document names are high-cardinality and often sensitive; they belong in protected traces if collected at all, not in metric labels [2].
How do I know whether an AI call actually succeeded?
Do not rely on the HTTP status. HTTP success is not necessarily task success; distinguish completed, canceled, timed out, rate-limited, unauthorized, invalid, upstream failure, parse failure, schema failure, tool failure, and business rejection [2].
What is the minimum viable tracing setup?
Instrumenting a single high-traffic endpoint with an SDK wrapper or OTLP exporter is enough to start capturing useful traces before expanding to full coverage [1].