A demo agent calls one tool and prints a happy answer. A production agent calls eight to twelve tools per turn, fans out, retries on three of them, falls back on one, and returns in 7 seconds. When something breaks at 2:14 AM, the difference between a 90-second triage and a 3-hour postmortem is what you logged on each of those spans. This post covers the span structure to ship, the fields each span should carry, how to redact PII and PHI without losing debug signal, and the OpenTelemetry semantic conventions for GenAI that are stabilizing as the cross-vendor floor.
LLM observability tools, the ones that ship with prompt-and-response logging out of the box, give you a single record per model call. That is enough when your system is one prompt, one completion, one response. It is not enough for an agentic system. An agentic turn is a tree, not a record. The user message triggers a planning step, the plan dispatches three tool calls, two succeed, one fails and retries twice, the retry succeeds, the agent re-plans with the new context, calls two more tools, then composes the final response. That is one user-visible turn and roughly fifteen logged events.
If you only capture the model calls, you cannot answer the questions that matter in production. Which tool was slow? Which tool failed and was retried? Did the planner choose the wrong tool on a known-good input? Did the response include data from a tenant that should not have been accessible? Those questions need the tool-call layer, not just the model layer. As LangChain frames it, agent observability is “the practice of monitoring, tracing, and debugging” the multi-step trajectories agents take, not just the input-and-output pairs of individual model calls. (LangChain, “AI Agent Observability”)
The architecture has to match the shape of the system. The shape of an agentic turn is a span tree.
A span is a unit of work with a start time, an end time, attributes, and a parent. OpenTelemetry’s tracing model is built for exactly this kind of tree, and the GenAI semantic conventions extend it for LLM-specific work. The pattern we ship for every agent in production:
OpenTelemetry’s GenAI conventions add a gen_ai.operation.name attribute on each span (chat, execute_tool, embeddings, generate_content) so a backend can roll up “all tool calls” or “all model calls” across an entire fleet without parsing free-form span names. (OpenTelemetry, “Semantic conventions for generative AI systems”) That is the contract worth following, because the major observability vendors (Datadog, Honeycomb, New Relic, Langfuse, Arize) are all converging on it. If you log to a custom schema you will pay the migration cost later.
A real span tree for a single intake-agent turn looks like this:
turn:req_2K8bZx4n (root, 6.8s, 3,418 tokens)
plan:opus-4-8 (840ms, 1,210 tokens)
tool:lookup_patient (320ms, ok)
tool:fetch_history (1.1s, ok, retries=1)
retry:fetch_history.attempt_2 (520ms, ok)
tool:check_allergies (180ms, ok)
tool:flag_review (90ms, ok)
compose:opus-4-8 (1.4s, 2,208 tokens)
Six tool calls, one retry, two model calls, one turn. Without the tree, you have six log lines that do not reference each other. With the tree, you have a debuggable trajectory.
A planning span and a tool span carry different fields, but both inherit a common envelope. The minimum we ship on every span in a production agent:
{
"trace_id": "tr_2026-05-29_a91f08b",
"span_id": "sp_lookup_patient_0001",
"parent_span_id": "sp_root_turn_0000",
"operation": "execute_tool",
"tool_name": "lookup_patient",
"agent_version": "intake-v3.2.1",
"tenant_id": "acme-clinic",
"user_id": "u_88431",
"start_ts": "2026-05-29T14:32:08.412Z",
"end_ts": "2026-05-29T14:32:08.732Z",
"latency_ms": 320,
"status": "ok",
"retries": 0,
"tool_args": {"mrn": "[REDACTED:phi.mrn]"},
"tool_output_hash": "sha256:9b4...",
"tool_output_size_bytes": 1842,
"model_revision": null,
"tokens_in": null,
"tokens_out": null,
"cost_usd_cents": 0
}
A planning or compose span adds the model fields and drops tool_name:
{
"operation": "chat",
"model": "claude-opus-4-8",
"model_revision": "anthropic-2026-05-15",
"prompt_template_id": "intake_plan_v9",
"tokens_in": 1210,
"tokens_out": 87,
"cost_usd_cents": 38,
"decision": "dispatch:lookup_patient,fetch_history,check_allergies,flag_review"
}
A few load-bearing notes:
tenant_id is mandatory, even on solo-tenant agents. Adding it later is a database migration; adding it now is one extra attribute. Multi-tenant SaaS needs it for per-customer attribution; single-tenant tools need it the day someone asks “did this run for staging or prod data?”agent_version and model_revision are not the same as model. “Claude Opus 4.8 on May 15” and “Claude Opus 4.8 on May 22” can behave differently. Capturing the revision lets you bisect a behavior regression to a model snapshot.prompt_template_id is versioned separately. Changing a prompt is a deploy, even if the weights are identical. The template id ties any decision to the instructions that produced it.tool_output_hash and tool_output_size_bytes let you correlate “the agent saw this output and then did X” without storing the full tool response in every log line. The full response goes to a separate object store keyed by hash.cost_usd_cents per span, summed at the root, builds a per-turn and per-tenant cost dashboard. Adding it after the fact requires backfilling token counts you may no longer have.Tool-call args are the most useful thing to log and the most dangerous to log raw. A lookup_patient call carries a medical record number. A send_email call carries an email address and a body. A process_refund call carries an account number and an amount. Unredacted, those turn your observability stack into a PHI repository.
The redaction pattern we ship:
PHI_FIELDS = {
"mrn", "ssn", "dob", "patient_name", "address", "phone",
"email", "account_number", "card_pan",
}
PHI_PATTERNS = [
(re.compile(r"bd{3}-d{2}-d{4}b"), "[REDACTED:phi.ssn]"),
(re.compile(r"bd{16}b"), "[REDACTED:phi.pan]"),
(re.compile(r"b[A-Z0-9._%+-]+@[A-Z0-9.-]+.[A-Z]{2,}b", re.I),
"[REDACTED:phi.email]"),
]
def redact(value):
if isinstance(value, dict):
return {
k: ("[REDACTED:phi." + k + "]" if k in PHI_FIELDS else redact(v))
for k, v in value.items()
}
if isinstance(value, list):
return [redact(v) for v in value]
if isinstance(value, str):
out = value
for pattern, replacement in PHI_PATTERNS:
out = pattern.sub(replacement, out)
return out
return value
def log_tool_span(tool_name: str, args: dict, **envelope):
return {
"operation": "execute_tool",
"tool_name": tool_name,
"tool_args": redact(args),
**envelope,
}
Two rules. First, redact at the logging boundary, not at the call site. If every tool author has to remember to redact, eventually one of them will not. Wrap the logger so redaction is automatic on every span. Second, log the shape of the redacted data, not just “[REDACTED]”. [REDACTED:phi.mrn] tells you the agent had an MRN in scope. [REDACTED] tells you nothing.
For HIPAA covered workflows, this redaction is part of the audit-controls obligation under 45 CFR §164.312(b). Observability log and audit log can share an envelope, but the audit log gets a stricter retention policy and a separate access boundary. We cover the audit-trail layer in the broader 5-layer production pattern.
A “the agent is slow” complaint is unanswerable without three numbers per turn:
Capture all three. Total turn time is the root span’s duration. TTFT needs an explicit marker on the moment the first response token leaves the agent process. TBT is captured by the streaming client and aggregated downstream.
The trap is treating “total agent latency” as a single number. A 6-second turn made of one slow tool call is fixable. A 6-second turn made of seven sequential 800ms tool calls needs parallelization. The difference shows up only when each tool span carries its own latency.
Most production agents call tools across process boundaries. A tool call hits a microservice, which hits a database, which calls a third-party API. If each layer logs to its own backend with no shared identifier, you cannot reconstruct a turn end to end. Stitching them together becomes a manual SQL exercise.
The fix is a correlation ID that travels with the call. OpenTelemetry’s W3C traceparent header is the standard: the agent generates a trace ID on the root span, propagates it via HTTP header on every outbound call, and every downstream service includes it on every log line and span. A single trace ID then retrieves every event across every service for a turn.
In a Python agent, that looks like:
from opentelemetry import trace, propagate
from opentelemetry.trace import SpanKind
tracer = trace.get_tracer("agent.intake")
def call_tool(tool_name: str, args: dict):
with tracer.start_as_current_span(
name=f"execute_tool.{tool_name}",
kind=SpanKind.CLIENT,
attributes={
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": tool_name,
"tenant.id": current_tenant_id(),
},
) as span:
headers = {}
propagate.inject(headers) # injects traceparent into the call
response = http_client.post(
tool_endpoint(tool_name),
json=redact(args),
headers=headers,
)
span.set_attribute("gen_ai.tool.response_size", len(response.content))
span.set_attribute("http.status_code", response.status_code)
return response
The downstream service does the same dance in reverse: extract the traceparent from inbound headers, attach it to every span and log line, propagate to its own outbound calls. Every event in the distributed call graph then carries the same trace ID, and Datadog, Honeycomb, or Langfuse renders the full tree in one view.
Token cost is the most underweighted observability dimension in agent systems. Sum the per-span costs along the tree, attribute them to a tenant and a user, roll up by hour and day, and you have a cost dashboard the finance team will not have to ask you to build later.
Per-span cost capture is straightforward arithmetic:
MODEL_PRICES = {
"claude-opus-4-8": {"in_per_mtok": 1500, "out_per_mtok": 7500},
"claude-sonnet-4-6": {"in_per_mtok": 300, "out_per_mtok": 1500},
"claude-haiku-4-5": {"in_per_mtok": 80, "out_per_mtok": 400},
# values in USD cents per million tokens; check provider pricing
# at deploy time and pin per environment
}
def span_cost_cents(model: str, tokens_in: int, tokens_out: int) -> int:
price = MODEL_PRICES[model]
return (
(tokens_in * price["in_per_mtok"]) // 1_000_000
+ (tokens_out * price["out_per_mtok"]) // 1_000_000
)
Two operational notes. First, pin the price table per environment and update it on a known schedule; budget alerts that hardcode old prices silently understate spend. Second, attribute cost to the tenant on the root span. A per-tenant cost dashboard is the difference between catching a runaway loop on one customer and finding it on the monthly bill.
A complete observability footprint fits in roughly a dozen attributes per span. What to skip: full prompt text, full tool args without redaction, full tool responses inline, raw PII in any form. Those go to a separately governed store accessed by hash when a debug session demands it. Logging everything to one place trades a small storage saving for a large compliance risk.
Anthropic’s production guidance: “extensive testing in sandboxed environments, along with the appropriate guardrails” is the gate to production. (Anthropic, “Building Effective Agents”) An eval harness tells you the agent should work. Observability tells you why it did not.
What should you log in an AI agent observability stack?
Per span: trace ID, span ID, parent span ID, operation type (chat or execute_tool), tool or model identity, agent and model version, tenant and user IDs, latency, status, retry count, redacted args, output hash and size, token counts, and per-span cost. Roll up at the root span for total turn time, total cost, and per-tenant attribution.
How do you redact PII and PHI from agent logs without losing debug signal?
Redact at the logging boundary, not at every call site. Maintain a list of sensitive field names and a list of regex patterns. Replace values with a typed placeholder like [REDACTED:phi.mrn] rather than a generic [REDACTED], so the log still tells you what kind of data was in scope. Store the response body in a separately governed store keyed by hash.
What is the difference between LLM observability and agent observability?
LLM observability captures one record per model call: prompt, response, latency, tokens. Agent observability captures the whole tree of a turn: planning, tool calls, retries, fallbacks, and the compose call. An agent turn is fifteen events, not one. If you only log model calls you cannot debug a slow or wrong tool call.
Should agent observability follow OpenTelemetry GenAI semantic conventions?
Yes. The OpenTelemetry GenAI conventions define gen_ai.operation.name, gen_ai.system, model attributes, and token attributes that the major backends (Datadog, Honeycomb, Langfuse, Arize) are converging on. A custom schema works short term and costs a migration later.
How do you track per-turn cost in an agentic system?
Capture input and output tokens on every model span. Maintain a pinned price table per environment. Compute per-span cost as (tokens_in * in_price + tokens_out * out_price) and sum to the root span. Attribute the root span to a tenant ID.
If your agent is in production or about to be, and you do not have answers to “what tool failed at 2 AM” and “which tenant is burning my token budget,” Mirgen and the Motomtech team have shipped this layer for HealthTech, FinTech, and multi-tenant SaaS deployments. Book a 15-min discovery call to walk through your current observability footprint and where the gaps are: https://cal.com/mirgen-motomtech/quick-intro.
Mirgen Hoxha, CEO, Motomtech.
