If you have ever deployed an AI agent into production and then watched it do something inexplicable — burn through tokens on a loop, call the wrong tool at the wrong time, or silently fail while the user sees a generic error — you already know the pain this article addresses. Traditional application monitoring tells you that a request arrived and a response left. It does not tell you why the agent chose one tool over another, where it spent three seconds deciding, or how much it cost to produce a single answer.

Agentic observability fills that gap. It is the practice of instrumenting, tracing, and analysing every decision an AI agent makes during a request lifecycle. For enterprise teams running multi-agent systems on Azure — particularly those of us building for regulated markets in Southeast Asia — it is not optional. It is the missing layer that determines whether your agent deployment is a controlled capability or an expensive black box.

This article walks through why observability matters for multi-agent systems, the specific signals you need to capture, how the emerging tooling landscape fits together, and how to implement the whole stack using Azure Monitor and Application Insights with minimal overhead.

Why Traditional Observability Falls Short

A standard microservices architecture emits three categories of telemetry: logs, metrics, and traces. Application Insights captures all three out of the box. You get request duration, dependency call success rates, exception stacks, and distributed trace correlation. That is excellent for HTTP APIs, message consumers, and database-backed services.

AI agents break this model in three ways.

First, agent execution is non-deterministic. The same user prompt can lead to different tool chains, different reasoning paths, and different resource consumption. A fixed trace template does not capture the branching logic that agents actually follow.

Second, agents make external calls that are invisible to standard dependency tracking. When an agent invokes a search API, a vector database, a code interpreter, or a third-party plugin, those calls may not go through standard HTTP clients that Application Insights auto-instruments. Without explicit custom dependency logging, you see gaps in the trace — the agent spent 200 milliseconds on "something," but you do not know what.

Third, cost is per-decision, not per-request. In a traditional API, cost correlates roughly with compute time. In an agent system, a single user question might trigger five LLM calls, three tool invocations, and two retry loops. The cost is additive and variable. Without per-step cost attribution, your monthly Azure bill becomes a mystery.

For enterprise teams — especially those handling sensitive data under PDPA or preparing for ISO 42001 compliance — these gaps are not just operational inconveniences. They are governance risks.

The Signals That Matter: What to Capture

Agentic observability is not about collecting everything. It is about collecting the right things. Here are the five signal categories that give you complete visibility into agent behaviour.

1. Agent Decision Traces

Every reasoning step an agent takes should emit a trace event. This includes the model call (which model, what prompt was sent, what tokens were used), the reasoning output (the agent's plan or decision), and the confidence or routing score if applicable.

In a multi-agent system, you also need to capture inter-agent communication. When Agent A delegates a subtask to Agent B, that handoff should appear as a parent-child span in your distributed trace. Application Insights supports this natively through Operation ID correlation.

2. Tool Call Monitoring

Each tool invocation — whether it is a database query, an API call, a file operation, or a code execution — should be logged as a custom dependency. You need:

  • Tool name and version
  • Input parameters (sanitised for PII)
  • Output summary (not full payload — store that in blob storage if needed)
  • Duration
  • Success or failure status
  • Token consumption if the tool itself calls an LLM

This gives you a complete map of what your agent actually did, not just what it intended to do.

3. Cost Attribution

Token usage is the primary cost driver. You need to track input tokens, output tokens, and the model tier for every LLM call. Multiply by Azure OpenAI pricing tiers and you have real-time cost attribution per request, per user, per agent, and per tool.

4. Latency Breakdown

End-to-end latency is misleading for agents. A 10-second response might be 800ms of network time, 200ms of tool execution, and 9 seconds of LLM inference. Breaking latency into these components tells you where to optimise. If 90% of your latency is LLM inference, caching or using a faster model for simple queries will have more impact than optimising your database queries.

5. Safety and Compliance Events

For enterprise deployments, you need to track every content filter trigger, every guardrail intervention, every PII detection event, and every human-in-the-loop escalation. These are not just operational metrics — they are audit evidence.

Where the Tooling Landscape Stands

Before building on Azure Monitor, it is worth knowing what the specialised market offers, because several dedicated LLM observability platforms have matured quickly.

Langfuse is the most prominent open-source option. It provides trace capture at the prompt and completion level, evaluation scoring, dataset management, and prompt versioning. Its recent v4 release moved to a ClickHouse backend for substantially faster ingestion, which matters once you are tracing thousands of agent turns a day.

LangSmith, from the LangChain team, positions itself as an end-to-end agent platform: observability, evaluation suites, and deployment tooling in one product. Arize Phoenix is the open-source tracing and evaluation layer from Arize AI, designed to pair with their production monitoring platform for teams that want ML-style observability over LLM workloads.

What these specialised tools add over a general APM is LLM-native depth: full prompt and response capture, evaluation scores computed with an LLM-as-a-judge, and side-by-side trace comparison across prompt versions.

The Azure-native path is different. Application Insights plus OpenTelemetry keeps every trace inside your existing governance, retention, and cost perimeter. The OpenTelemetry GenAI semantic conventions now standardise the span and attribute names for LLM calls — including dedicated agent spans and an OpenAI instrumentation — so your instrumentation stays vendor-neutral and portable.

The trade-off is straightforward. Specialised tools give you a faster developer loop and richer evaluation features, but they are another vendor and another data perimeter — and remember that agent traces contain prompts, which in regulated industries may carry customer data. Azure-native gives you unified alerting, workbooks, KQL, and role-based access with no extra data egress, but you build the LLM-specific views yourself.

My recommendation for enterprise teams already running Azure Monitor: start there. Add a specialised tool later if evaluation-heavy workflows justify the additional data perimeter.

Practical Implementation with Azure Monitor and Application Insights

Let me walk through a concrete implementation. This is the pattern I use with enterprise clients running agents on Azure Container Apps or Azure Kubernetes Service.

Step 1: Instrument Your Agent Framework

Whether you are using Semantic Kernel, AutoGen, LangChain, or a custom framework, you need a telemetry wrapper around every agent action. Here is a Python example using the Application Insights SDK:

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Configure Application Insights
configure_azure_monitor(
    connection_string="InstrumentationKey=your-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/"
)

tracer = trace.get_tracer("agent-service")

def execute_agent_turn(user_prompt: str, agent_id: str):
    with tracer.start_as_current_span("agent_turn") as span:
        span.set_attribute("agent.id", agent_id)
        span.set_attribute("user.prompt_length", len(user_prompt))
        
        # Log the reasoning step
        with tracer.start_as_current_span("reasoning") as reasoning_span:
            model_response = call_llm(agent_id, user_prompt)
            reasoning_span.set_attribute("model.name", model_response.model)
            reasoning_span.set_attribute("tokens.input", model_response.usage.prompt_tokens)
            reasoning_span.set_attribute("tokens.output", model_response.usage.completion_tokens)
            reasoning_span.set_attribute("cost.usd", calculate_cost(model_response))
        
        # Log the tool call as a custom dependency
        if model_response.tool_call:
            with tracer.start_as_current_span("tool_call") as tool_span:
                tool_span.set_attribute("tool.name", model_response.tool_call.name)
                tool_result = execute_tool(model_response.tool_call)
                tool_span.set_attribute("tool.duration_ms", tool_result.duration_ms)
                tool_span.set_attribute("tool.success", tool_result.success)
        
        return build_response(model_response)

To provision the Application Insights workspace itself as infrastructure-as-code, the Bicep looks like this:

param location string = resourceGroup().location
param workspaceName string = 'law-agent-observability'
param appInsightsName string = 'appi-agent-observability'

resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2025-07-01' = {
  name: workspaceName
  location: location
  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: 90
  }
}

resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: appInsightsName
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: logAnalytics.id
    IngestionMode: 'LogAnalytics'
  }
}

output connectionString string = appInsights.properties.ConnectionString

Two details matter here. IngestionMode: 'LogAnalytics' routes telemetry through Log Analytics, which is what unlocks KQL queries and log-based metrics across your whole platform. And the retentionInDays setting is your first cost lever — agent traces are verbose, so 90 days is a reasonable starting point for interactive queries, with longer retention exported to cheap blob storage if compliance requires it.

Step 2: Set Up Custom Metrics in Application Insights

Standard metrics (request rate, duration, failures) appear automatically. For agent-specific metrics, emit them through OpenTelemetry (the same exporter already configured for traces):

from opentelemetry import metrics

meter = metrics.get_meter("agent-service")

tokens_counter = meter.create_counter(
    "agent.tokens.total",
    description="Total tokens consumed per agent",
)
cost_counter = meter.create_counter(
    "agent.cost.usd",
    description="Estimated cost per agent turn",
)
llm_latency = meter.create_histogram(
    "agent.latency.llm_ms",
    unit="ms",
)

# Record per LLM call, with agent/model dimensions
tokens_counter.add(total_tokens, {
    "agent_id": agent_id,
    "model": model_name,
})
cost_counter.add(cost_usd, {
    "agent_id": agent_id,
})
llm_latency.record(llm_latency_ms, {"agent_id": agent_id})

These custom metrics land in Application Insights alongside your standard metrics and can be queried with KQL.

Step 3: Build a Workbooks Dashboard

Azure Monitor Workbooks give you rich, shareable dashboards. Create a workbook with these sections:

  • Agent Health: Request rate, success rate, and p95 latency by agent ID
  • Cost Overview: Daily token consumption and estimated cost, broken down by model tier and agent
  • Tool Usage: Heatmap of tool invocations, success rates, and average duration
  • Error Analysis: Top errors by agent, with drill-down to trace details
  • Safety Events: Content filter triggers, guardrail interventions, and PII detections over time

Step 4: Set Up Alerts

Configure alert rules for the metrics that matter:

# Alert on high token consumption (potential runaway agent)
az monitor metrics alert create \
  --name "agent-high-token-usage" \
  --resource-group my-rg \
  --scopes "/subscriptions/sub-id/resourceGroups/my-rg/providers/microsoft.insights/components/my-app-insights" \
  --condition "total agent.tokens.total > 50000" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action my-action-group

# Alert on elevated failed-request volume (requests/failed is the standard
# Application Insights failed-request metric — it is a count, so use a
# total aggregation, not a percentage)
az monitor metrics alert create \
  --name "agent-error-rate" \
  --resource-group my-rg \
  --scopes "/subscriptions/sub-id/resourceGroups/my-rg/providers/microsoft.insights/components/my-app-insights" \
  --condition "total requests/failed > 25" \
  --window-size 10m \
  --action my-action-group

Step 5: Implement Distributed Tracing Across Multi-Agent Systems

When multiple agents collaborate, you need trace context propagation. The OpenTelemetry standard handles this. Pass the trace context through your agent messaging layer:

from opentelemetry.propagate import inject

def delegate_to_specialist(task, parent_context):
    headers = {}
    inject(headers, context=parent_context)
    
    # Send task with trace headers to specialist agent
    response = requests.post(
        "https://specialist-agent.internal/task",
        json=task,
        headers=headers
    )
    return response

This ensures that when you view a trace in Application Insights, you see the complete journey across all agents as a single correlated operation.

Cost Tracking: Making AI Spend Visible

Cost management deserves special attention because it is often the first executive concern. Here is the pattern that works.

Every LLM call emits a trace event with token counts and model tier. A background function aggregates these into hourly cost summaries written to Log Analytics. A Power BI dashboard — or an Azure Monitor Workbook — surfaces the data.

The key metrics to track:

  • Cost per agent: Which agents are the most expensive?
  • Cost per user: Are power users driving disproportionate spend?
  • Cost per task type: Is the research agent spending 10x more than the triage agent? Is that justified?
  • Cost trend: Are costs growing linearly with usage, or is there a non-linear component (e.g., retry loops) driving costs up?

For Southeast Asian enterprises billing in MYR, SGD, or THB, you can extend the aggregation to apply real-time exchange rates. This makes cost discussions with finance teams much more productive.

Latency Analysis: Where Does the Time Go?

Agent latency is a user experience killer. A 15-second response feels broken, even if it is producing a high-quality answer. Here is how to diagnose and fix latency.

Instrument every step with start and end timestamps. In Application Insights, this gives you a waterfall view of each agent turn. Common findings:

  • LLM inference dominates (70-90% of total latency): Use streaming responses to show partial results. Consider a smaller, faster model tier — gpt-5-nano or gpt-5-mini on Azure OpenAI — for simpler subtasks within your agent pipeline.
  • Tool calls are slow (databases, external APIs): Add caching layers. Pre-fetch common data. Use semantic caching for repeated queries.
  • Agent-to-agent communication adds up (multi-agent systems): Reduce the number of agent hops. Use direct tool calls instead of delegating to specialist agents when the task is simple.
  • Retry loops inflate latency: Set explicit timeout and retry budgets. Log retry counts so you can identify problematic patterns.

A Note on Compliance and Governance

For enterprise deployments in regulated industries — financial services, healthcare, government — agentic observability is also a compliance enabler. PDPA in Malaysia, PDPA in Singapore, and emerging AI governance frameworks across ASEAN all point toward the same direction: you need to demonstrate control over automated decision-making systems.

Detailed trace logs showing exactly what data an agent accessed, what decisions it made, and what outputs it produced are the foundation of an AI audit trail. Store traces in immutable storage (Azure Blob with WORM policy or Azure Immutable Blob Storage) alongside your Application Insights data for long-term retention.

Pitfalls I See Repeatedly

A few failure patterns show up in almost every agent observability implementation I review.

Pitfall 1: Logging full prompts and responses by default. It feels thorough, but agent traces containing prompts can carry customer data straight into your telemetry pipeline. That creates a second copy of personal data with its own access surface, retention clock, and compliance exposure. Sanitise or truncate by default, and gate full-payload capture behind an explicit flag with its own access control.

Pitfall 2: Telemetry volume surprises. Agent traces are an order of magnitude more verbose than HTTP service traces — every LLM call, every tool invocation, every handoff emits spans and custom dimensions. At Log Analytics ingestion pricing, an uninstrumented-for-volume rollout can quietly double your monitoring bill. Set daily caps, sample verbose attributes, and review ingestion volume after the first week, not the first month.

Pitfall 3: Alerting on averages. Average token consumption and average latency hide the exact behaviour you care about — the runaway loop, the retry storm, the single agent turn that cost ten times normal. Alert on totals and p95/p99 percentiles, and use per-agent dimensions so one misbehaving agent does not get masked by nine healthy ones.

Pitfall 4: Treating observability as an afterthought. Teams that add tracing after their first production incident end up retrofitting instrumentation into framework internals. The cost of instrumenting from day one is a few wrapper functions per agent action. The cost of not having traces during your first incident is measured in hours of blind debugging.

Pitfall 5: Cost attribution without a denominator. Tracking total spend is easy; making it actionable requires dividing by something — cost per resolved ticket, per completed task, per active user. Without a business denominator, cost conversations stay abstract and budgets get cut instead of steered.

Key Takeaways

  1. Agentic observability is non-negotiable for production AI agents. Traditional monitoring captures the request-response cycle but misses the reasoning, tool calls, and cost dynamics that define agent behaviour.
  1. Instrument every decision, tool call, and LLM invocation. Use OpenTelemetry and Application Insights custom spans and dependencies to build a complete picture of what your agent does and why.
  1. Cost tracking must be per-step, not per-request. Token consumption varies wildly between agent turns. Per-step attribution is the only way to manage AI spend responsibly.
  1. Latency analysis reveals where to optimise. Waterfall traces that break down LLM inference time, tool execution, and inter-agent communication tell you exactly which component to tune.
  1. Start with Azure Monitor and Application Insights. They are already in your Azure subscription, they scale with your agent deployment, and they integrate with the rest of your Azure governance stack. No additional vendor needed.

Have questions about implementing agentic observability in your Azure environment? Reach out at wenfeng.my or find me on LinkedIn. I work with enterprise teams across Southeast Asia on production AI agent architectures.