LLMOps: Instrumenting Your Multi-Agent Stack for Production

The Observability Gap in Agentic Systems

Traditional software observability is built on a simple model: a request comes in, some deterministic code runs, a response goes out. You log inputs and outputs, track latency, alert on errors. This model breaks when you introduce agents.

Agentic systems have four properties that make conventional logging insufficient:

For an agentic system, you need three layers:

As of 2026, the industry has largely converged on OpenTelemetry as the portable instrumentation standard for the trace layer. The semantic conventions for LLM spans are stabilizing, and every major LLMOps platform (Langfuse, Braintrust, Arize Phoenix) ingests OTEL traces natively. You instrument once and route to any backend.

Most teams alert on exceptions and HTTP 5xx errors. In agentic systems, the real failures are often silent:

• An agent that completes successfully but gives the wrong answer
• An agent that takes 45 seconds instead of 4 because it got into a retry loop
• An agent that's routing all requests to Opus when most could run on Haiku
• A cache hit rate that dropped from 80% to 20% after a prompt change

None of these show up as errors. They show up as cost spikes, latency degradation, and eventually as user complaints. Module 4 covers what to actually alert on.

Tracing Agent Calls with OpenTelemetry

OpenTelemetry (OTEL) is the CNCF standard for distributed tracing. It's the right choice for LLMOps instrumentation because it's backend-agnostic — you write instrumentation once and export to Langfuse, Braintrust, Jaeger, Datadog, or any other OTEL-compatible backend. No vendor lock-in at the instrumentation layer.

• The overall agent run (root span)
• Each LLM call
• Each tool invocation
• Each sub-agent handoff
• Retries (as child spans of the failed span)

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("my-agent")

def run_agent(user_message: str, user_id: str, feature: str) -> str:
    with tracer.start_as_current_span("agent.run") as root_span:
        root_span.set_attributes({
            "agent.user_id": user_id,
            "agent.feature": feature,
            "agent.input_length": len(user_message),
        })
        
        result = agent_loop(user_message)
        
        root_span.set_attributes({
            "agent.output_length": len(result),
            "agent.steps_taken": agent_step_count,
        })
        return result

def call_llm(prompt: str, model: str) -> dict:
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attributes({
            "llm.model": model,
            "llm.input_tokens": count_tokens(prompt),
            "llm.request_type": "chat",
        })
        
        response = client.messages.create(
            model=model, max_tokens=1024,
            messages=[{"role": "user", "content": prompt}]
        )
        
        span.set_attributes({
            "llm.output_tokens": response.usage.output_tokens,
            "llm.cache_read_tokens": getattr(response.usage, 'cache_read_input_tokens', 0),
            "llm.stop_reason": response.stop_reason,
        })
        return response

def invoke_tool(tool_name: str, tool_input: dict) -> dict:
    with tracer.start_as_current_span("tool.call") as span:
        span.set_attributes({
            "tool.name": tool_name,
            "tool.input_size": len(str(tool_input)),
        })
        result = execute_tool(tool_name, tool_input)
        span.set_attributes({
            "tool.result_size": len(str(result)),
            "tool.success": result.get("success", True),
        })
        return result

When your orchestrator calls a sub-agent over HTTP (or A2A), inject the trace context into the request headers so the sub-agent's spans attach to the parent trace:

# Orchestrator: inject context into outgoing request
from opentelemetry.propagate import inject

headers = {}
inject(headers)  # adds traceparent, tracestate headers
response = httpx.post(specialist_agent_url, json=payload, headers=headers)

# Sub-agent: extract and continue the trace
from opentelemetry.propagate import extract

def handle_request(request):
    ctx = extract(request.headers)
    with tracer.start_as_current_span("sub-agent.run", context=ctx):
        # this span links to the orchestrator's trace
        ...

This is what gives you end-to-end visibility across a multi-agent call tree — every span, from every agent, in one trace.

The OpenTelemetry community is standardizing LLM span attributes under the gen_ai.* namespace. Key attributes to follow:

• gen_ai.system — the provider (e.g. anthropic)
• gen_ai.request.model — the requested model
• gen_ai.response.model — the actual model used
• gen_ai.usage.input_tokens / gen_ai.usage.output_tokens
• gen_ai.operation.name — chat, text_completion, embeddings

Using standard attribute names means your traces are immediately readable by any OTEL-native dashboard without custom parsing.

Tools: Langfuse, Braintrust, and Weights & Biases

Once you're emitting OTEL traces, you need a backend to store, visualize, and evaluate them. Three platforms dominate indie dev and small team usage in 2026. The right choice depends on your priorities, not brand preference.

Langfuse is MIT-licensed, has 19,000+ GitHub stars, and can be self-hosted with a single Docker Compose file. It ingests OTEL natively and also provides its own SDK with a @observe decorator:

from langfuse.decorators import observe, langfuse_context

@observe()  # automatically traces this function as a span
def run_agent(user_message: str) -> str:
    langfuse_context.update_current_observation(
        input=user_message,
        metadata={"user_tier": "pro"}
    )
    result = agent_loop(user_message)
    langfuse_context.update_current_observation(output=result)
    return result

Langfuse's SQL export via BigQuery integration lets data science teams run custom analysis on trace data — cost breakdowns by feature, quality trends over time, outlier detection.

Braintrust's distinctive feature is closing the production-to-test loop: production traces automatically become test cases. When a user-facing interaction fails (or is flagged by LLM-as-judge), it lands in your eval dataset. Your next deployment is benchmarked against it automatically.

Integration via SDK, OTEL, or the AI Proxy (a drop-in proxy layer that intercepts LLM calls with zero code changes):

import braintrust

# AI Proxy: just change the base URL, zero other changes
client = anthropic.Anthropic(
    base_url="https://braintrustproxy.com/v1",
    api_key=braintrust.api_key()
)
# Every subsequent call is automatically logged and traceable

W&B added LLM tracing to Weave, its data management platform. If you're already using W&B for training runs and model evaluation, Weave integrates naturally.

import weave

weave.init("my-agent-project")

@weave.op()  # traces this function
def call_agent(prompt: str) -> str:
    return run_agent(prompt)

| Priority | Tool |

|---|---|

| Self-host / data residency | Langfuse |

| Eval-regression loop | Braintrust |

| Existing W&B / ML stack | Weights & Biases |

| Framework-agnostic, OTEL-first | Langfuse |

| Minimal setup, proxy integration | Braintrust |

| Budget: $0 cloud | Langfuse (self-host) |

For new projects with no existing tooling: start with Langfuse self-hosted. The OTEL-native instrumentation you write works with any backend, so switching costs are low if your needs change.

Alerting on Agent Failures: Latency, Cost, and Correctness

Alerting on agentic systems is different from alerting on APIs. The failure modes are subtler, the signals are distributed, and "error rate" as a metric misses most of what actually goes wrong. Here's what to actually alert on.

Agent latency is bimodal by design — fast runs (happy path) and slow runs (retries, tool failures, long reasoning chains). The metric you want is p99 latency by feature, not mean latency.

Specific alerts worth setting:

• Step count outlier: Alert when an agent run exceeds 2x the median step count for that task type. This catches infinite loops and pathological retry behavior before they drain cost budgets.
• Total run duration > threshold by task type: Different tasks have different expected durations. A research task at 90 seconds is normal; a classification task at 90 seconds is a bug.
• Sub-agent timeout rate: If your A2A sub-agents are timing out more than 1% of the time, something upstream is broken — wrong input format, overloaded specialist, auth failure.

Cost alerts are your circuit breaker for agentic systems. Three specific patterns:

Latency and cost are measurable from observability data. Correctness requires evaluation. Three practical approaches, from lightest to heaviest:

from jsonschema import validate, ValidationError

def validate_agent_output(output: dict, schema: dict) -> bool:
    try:
        validate(instance=output, schema=schema)
        return True
    except ValidationError as e:
        metrics.increment("agent.output.schema_error", tags={"field": e.path})
        return False

def score_output(input_text: str, output_text: str) -> float:
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        system="Score this agent output 0.0-1.0 for correctness and helpfulness. Respond with only the number.",
        messages=[{"role": "user", "content": f"Input: {input_text}\n\nOutput: {output_text}"}]
    )
    return float(response.content[0].text.strip())

• Agent error rate above 10% in a 5-minute window
• Token burn rate 5x daily average (loop or routing catastrophe)
• Complete sub-agent unavailability (100% timeout rate)

• Cost per outcome drifting 25% over 7-day baseline
• Cache hit rate below 40% (prompt change broke caching)
• p99 latency above 2x baseline for any feature

• LLM judge quality scores by feature and model
• Model distribution shift (requests migrating up-tier unexpectedly?)
• Schema validation failure rate trends

The goal isn't to be paged on everything — it's to catch the class of failures that are invisible to error-rate monitoring but visible to users and your finance team.