Evaluation Framework

Overview

The Enterprise Kit includes a comprehensive evaluation framework powered by the Azure AI Foundry Evaluations SDK (azure-ai-evaluation). It provides built-in evaluators, custom evaluator support, evaluation datasets, and CI/CD integration. Evaluations run continuously via Azure AI Foundry Continuous Evaluation (GA) — with scheduled sampling, cron-based execution, and results flowing to Azure Application Insights and Azure Cosmos DB for trend analysis.

Built-in Evaluators

The eval-packs/builtin/ directory contains pre-configured evaluators based on the Azure AI Foundry evaluation SDK (azure-ai-evaluation):

Agent Process Evaluators (Foundry-Native)

Process evaluators are classified as warning-only for release gates, allowing teams to establish baselines before enforcing hard thresholds. They are especially valuable for operations-agent and workflow-agent classes that heavily depend on tool integrations.

Process Evaluator Data Flow

Agent Run (with tool calls)
        │
        ├── tool_calls[]        → actual tool invocations captured via OTel spans
        ├── tool_definitions[]  → available tool schemas from agent manifest
        └── expected_tool_calls[] → ground-truth tool call sequence from eval dataset
        │
        ▼
Process Evaluators (Azure AI Foundry Agent Evaluator SDK)
        │
        ├── tool_call_accuracy      → composite sequence correctness
        ├── tool_selection          → precision/recall of tool choice
        ├── tool_input_accuracy     → parameter value correctness
        ├── tool_output_utilization → response-to-output alignment
        └── tool_call_success       → runtime success & error recovery
        │
        ▼
Scorecard Compiler → Release Gate (warning_only)
        │
        └── Grafana "Tool Call Quality" Dashboard

Content Safety Evaluators (Azure AI Foundry)

Additional evaluators powered by Azure AI Content Safety:

Custom Evaluators

Two enterprise-specific custom evaluators are included in eval-packs/custom/:

Business Intent Evaluator

Measures whether the agent's response correctly addresses the business intent behind the user's query — not just lexical similarity but semantic alignment with business objectives.

Policy Compliance Evaluator

Checks whether agent responses comply with configured policy packs — data boundary restrictions, PII handling, escalation rules, and tool usage constraints.

Evaluation Datasets

Datasets in eval-packs/datasets/ contain query-response pairs with ground-truth annotations. Create domain-specific datasets following this format:

# eval-packs/datasets/my-domain.jsonl
{"query": "What is our refund policy?", "ground_truth": "30-day refund...", "context": "..."}
{"query": "How do I reset MFA?", "ground_truth": "Contact IT help desk...", "context": "..."}

Running Evaluations

CLI

python sdk-starters/python/eval_runner.py \
  --manifest agent-templates/knowledge-agent/agent-manifest.yaml \
  --eval-pack eval-packs/builtin/builtin-evaluators.yaml \
  --dataset eval-packs/datasets/my-domain.jsonl \
  --output results/eval-report.json

CI/CD Integration

The eval-packs/ci-wiring/eval-ci-integration.py module integrates with GitHub Actions and Azure DevOps pipelines:

• Runs eval suite on every PR
• Blocks merge if trust score drops below threshold
• Posts eval summary as PR comment
• Archives results to Azure Cosmos DB for trend analysis

Continuous Evaluation (Azure AI Foundry GA)

Beyond CI/CD gates, the framework leverages Azure AI Foundry Continuous Evaluation (GA) for scheduled monitoring of production agents:

1. Define eval schedule in agent manifest (evaluation.schedule)
2. Azure AI Foundry runs evaluations with live traffic sampling
3. Results feed into Scorecard Compiler for trust score updates
4. Alerts triggered via Azure Application Insights if scores drop below baseline
5. Remediation workflows auto-triggered via Azure Monitor action groups

Eval Results Schema

{
  "agent_id": "knowledge-agent-v1.0.0",
  "eval_pack": "builtin",
  "timestamp": "2025-01-15T10:30:00Z",
  "metrics": {
    "groundedness": 0.92,
    "relevance": 0.88,
    "coherence": 0.95,
    "business_intent": 0.85,
    "policy_compliance": 1.0,
    "tool_call_accuracy": 0.82,
    "tool_selection": 0.90,
    "tool_input_accuracy": 0.78,
    "tool_output_utilization": 0.85,
    "tool_call_success": 0.95
  },
  "trust_score": 0.91,
  "dataset_size": 150,
  "pass": true,
  "threshold": 0.8
}

Process Evaluator Dataset Format

Entries targeting process evaluators require additional fields in the evaluation dataset (eval-packs/datasets/):

{
  "id": "golden-006",
  "agent": "operations-agent",
  "query": "Create an incident for the payment processing outage",
  "expected_answer": "I'll create a P1 incident ticket in ServiceNow...",
  "context": "ServiceNow ITSM integration active.",
  "tool_definitions": [
    {"name": "servicenow_create_incident", "description": "Create incident", "parameters": {...}}
  ],
  "expected_tool_calls": [
    {"name": "servicenow_create_incident", "arguments": {"priority": "P1"}}
  ],
  "evaluators": ["task_adherence", "tool_call_accuracy", "tool_selection",
                  "tool_input_accuracy", "tool_output_utilization", "tool_call_success"]
}