IBM's Azure Foundry Enterprise Kit

Operations Guide

Day-2 operations for AI agent platforms. This guide covers health monitoring, incident response, scaling, backup, secret rotation, agent lifecycle management, cost management, compliance, disaster recovery, and runbook automation — aligned to WAF Operational Excellence (OE:07, OE:10) and NIST AI RMF (MANAGE 2.2).

Table of Contents

  1. Day-2 Operations Overview
  2. Health Monitoring
  3. Incident Response
  4. Scaling Procedures
  5. Backup & Restore
  6. Secret & Certificate Rotation
  7. Agent Lifecycle Management
  8. Cost Management
  9. Compliance & Audit
  10. Disaster Recovery
  11. Troubleshooting Playbooks
  12. Runbook Automation

1. Platform Components

ComponentAzure ServiceHealth SignalSLA
AI ModelsAzure FoundryModel deployment status99.9%
Agent MemoryCosmos DBAvailability metric99.99%
Decision LedgerCosmos DBAvailability metric99.99%
Search IndexAI SearchService status99.9%
SecretsKey VaultService health99.99%
TelemetryApp InsightsIngestion latency99.9%
AI GatewayAPI ManagementGateway health99.95%
NetworkingVNet / PEsResource health99.99%

Operational Responsibility Model

Platform Team           Agent Developers          Security Team
├── Infrastructure      ├── Agent manifests       ├── Policy packs
├── Networking          ├── Prompt engineering     ├── Cert pipeline
├── Monitoring          ├── Toolbox configs        ├── Audit reviews
├── Scaling             ├── Eval datasets          ├── Incident triage
├── DR procedures       ├── SDK integration        ├── Compliance
└── Cost governance     └── Agent certification    └── Trust scores

2. Health Monitoring

Critical Metrics

MetricSourceThresholdAlert Severity
Agent response latency (P95)App Insights> 30sSev 2
Agent error rateApp Insights> 5%Sev 1
Trust score averageCosmos DB< 0.7Sev 2
Token consumption rateAPIM> 80% budgetSev 3
Cosmos DB RU consumptionCosmos DB> 80% provisionedSev 2
Key Vault throttlingKey Vault> 0 events/5minSev 3
Decision ledger write failuresApp Insights> 0Sev 1
Policy violation countApp Insights> 0 (block mode)Sev 1

Alert Escalation

Sev 3 (Warning)   → Slack/Teams notification → 4h response
Sev 2 (Error)     → PagerDuty/on-call page   → 1h response
Sev 1 (Critical)  → PagerDuty P1 + bridge    → 15min response
Sev 0 (Outage)    → Incident Commander        → Immediate

3. Incident Response

Severity Classification

SeverityImpactExampleResponse Time
Sev 0Full platform outageFoundry endpoint unreachableImmediate
Sev 1Agent degradationAll agents returning errors15 min
Sev 2Partial degradationSingle agent class failing1 hour
Sev 3Non-critical issueElevated latency, warnings4 hours
Sev 4InformationalMinor config drift detectedNext business day

Response Procedure

1. DETECT    → Automated alert or user report
2. TRIAGE    → Classify severity, assign owner
3. CONTAIN   → Isolate affected agent/service
4. DIAGNOSE  → Use troubleshooting playbooks (Section 11)
5. RESOLVE   → Apply fix, verify in staging first
6. VERIFY    → Run eval pipeline against affected agent
7. POSTMORTEM → Document root cause, update runbooks

Escalation Matrix

ConditionEscalation TargetAction
Agent trust score < 0.5Security TeamDisable agent, review ledger
Policy violation (block)Compliance OfficerReview, approve/reject
Model endpoint downPlatform TeamFailover to secondary model
Data boundary violationSecurity + LegalImmediate containment
Cost spike > 200%FinOps + PlatformBudget alert, throttle

4. Scaling Procedures

Cosmos DB (Serverless): Auto-scales to 5000 RU/s. No action needed below that threshold. If exceeding limits, convert to provisioned throughput (requires data export/import).

AI Search: Scale replicas for throughput, partitions for index size. Use az search service update to adjust.

API Management: Scale units for gateway throughput. Premium tier supports multi-region deployment for HA.

5. Proactive Health Checks

Run weekly:

# Validate all configurations
python scripts/validate_all.py

# Check Key Vault expiring secrets (< 30 days)
az keyvault secret list --vault-name kv-foundry-prod \
  --query "[?attributes.expires < '<30d-from-now>']" \
  -o table

# Verify Azure Policy compliance
az policy state summarize \
  --resource-group rg-foundry-prod \
  --query "policyAssignments[?results.nonCompliantResources > `0`]" \
  -o table

Related Resources