Dashboards & Monitoring
Day-2 visibility out of the box. Pre-built Azure Monitor Workbooks give operations teams immediate insight into agent health, performance, guardrail effectiveness, and compliance posture — no dashboard building required.
Available Dashboards
| Dashboard | Format | KPIs Tracked |
|---|---|---|
| Agent Operations | Azure Monitor Workbook | Request volume, latency, errors, guardrails, tokens, evals |
Dashboard KPIs
The Agent Operations Dashboard tracks 10 key metrics:
| # | KPI | What It Measures | Alert Threshold |
|---|---|---|---|
| 1 | Request Volume | Agent invocations per hour | Spike > 3x baseline |
| 2 | Latency P50/P95/P99 | Response time percentiles | P95 > 10s |
| 3 | Guardrail Trigger Rate | % of runs triggering content safety | > 5% |
| 4 | Token Usage | Prompt + completion tokens per agent | Daily budget exceeded |
| 5 | Evaluation Scores | Groundedness, safety, coherence trends | Score drop > 10% |
| 6 | HITL Escalation Rate | Human-in-the-loop frequency | > 15% of runs |
| 7 | Interceptor Actions | Kill switch, blocks, audit decisions | Any kill switch activation |
| 8 | Error Rate | Failed agent runs by error type | > 1% |
| 9 | Decision Ledger | Trust decisions recorded | Audit trail gaps |
| 10 | Agent Health Summary | Composite health per agent | Any “Unhealthy” agent |
Setup
Option 1: Azure Monitor Workbook (Recommended)
- Open Azure Monitor → Workbooks → New
- Click Advanced Editor (code icon
</>) - Paste the contents of
dashboards/agent-operations-dashboard.json - Update the Application Insights resource reference
- Click Apply → Save
Option 2: Import via Azure CLI
# Set your Application Insights resource ID
AI_RESOURCE_ID="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Insights/components/<name>"
# Create the workbook
az monitor app-insights workbook create \
--resource-group <rg> \
--display-name "Agent Operations Dashboard" \
--serialized-data @dashboards/agent-operations-dashboard.json \
--source-id "$AI_RESOURCE_ID" \
--category "AI Agent Monitoring"Option 3: KQL Queries (Ad-Hoc)
Individual KQL queries are available in observability/kql/agent-monitoring-queries.kql. Run them directly in the Application Insights Logs blade.
Customization
Adding Agent-Specific Panels
Add a new query item to the dashboard JSON:
{
"type": "query",
"title": "My Custom KPI",
"query": "AppDependencies\n| where Name == \"agent.run\"\n| extend AgentName = tostring(customDimensions[\"agent.name\"])\n| where AgentName == \"my-agent\"\n| summarize Count = count() by bin(TimeGenerated, 1h)\n| render timechart",
"visualization": "timechart"
}Alert Rules
Configure alerts in infra/bicep/modules/app-insights.bicep using the action groups defined for your environment. Production alert email recipients are set in infra/bicep/parameters/prod.bicepparam.
Related Resources
- Telemetry Schema — OTel span and attribute definitions
- Operations Guide — Day-2 operational procedures
- Deployment Guide — Infrastructure setup including App Insights