---
title: "Eval-Gated Pipeline Promotion"
description: "Block pipeline progression until eval scenarios pass and prevent regressions"
---

# Eval-Gated Pipeline Promotion

Eval-gated pipelines let you enforce quality gates before code progresses through your CI/CD workflow. Run Cage Eval scenarios as pipeline stages and automatically block promotion when tests fail or regress.

## Overview

An eval gate is a pipeline stage that:
1. Runs one or more eval scenarios against your agent
2. Compares results against configurable thresholds
3. Detects regressions vs. historical baselines
4. Blocks or allows pipeline progression based on results

## Stage Types

### `eval` Stage

Runs a Cage Eval scenario or suite within the pipeline.

```yaml
stages:
  - name: run-tests
    type: eval
    config:
      scenario_name: "fix-bug-scenario"  # OR scenario_id: uuid
      min_pass_rate: 0.8                  # 80% assertions must pass
      max_cost_usd: 0.50                  # Budget cap
      fail_on_regression: true
      baseline_count: 5                   # Compare against last 5 runs
      regression_threshold: 0.1           # 10% max score drop
```

**Configuration options:**

| Field | Type | Description |
|-------|------|-------------|
| `scenario_id` | UUID | ID of the scenario to run |
| `scenario_name` | string | Name of the scenario (alternative to ID) |
| `suite_id` | UUID | Run all scenarios in a suite |
| `suite_name` | string | Suite name (alternative to ID) |
| `min_pass_rate` | float | Minimum pass rate (0-1) to pass gate |
| `max_cost_usd` | float | Maximum allowed cost for the eval run |
| `fail_on_regression` | bool | Block if score regresses vs baseline |
| `baseline_count` | int | Number of previous runs for baseline (default: 5) |
| `regression_threshold` | float | Max allowed score drop (default: 0.1 = 10%) |
| `store_result_in_state` | bool | Save eval result to pipeline state |
| `result_state_key` | string | State key for result (default: "eval_result") |

### `gate` Stage

Evaluates trust score and cost thresholds.

```yaml
stages:
  - name: quality-gate
    type: gate
    config:
      trust_above: 70      # Session trust score >= 70
      cost_below: 1.00     # Total run cost < $1.00
```

**Configuration options:**

| Field | Type | Description |
|-------|------|-------------|
| `trust_above` | int | Minimum trust score (0-100) to pass |
| `cost_below` | float | Maximum cost in USD to pass |
| `expression` | string | Custom CEL expression (coming soon) |
| `state_key` | string | State key for dynamic evaluation |
| `fail_fast` | bool | Stop pipeline immediately on failure |

## Regression Detection

When `fail_on_regression: true` is set, the eval gate:

1. Fetches the last N passing runs (configurable via `baseline_count`)
2. Calculates the baseline average score
3. Compares the current run's score against the baseline
4. Fails if the score dropped more than `regression_threshold`

**Example:**
- Baseline (last 5 runs): 95%, 92%, 98%, 94%, 96% → **average: 95%**
- Current run: 82%
- Delta: -13% (exceeds 10% threshold)
- **Result: Gate fails with regression error**

## Pipeline Example

```yaml
name: agent-ci
version: 1

stages:
  - name: build
    type: command
    command: "npm run build"

  - name: unit-tests
    type: command
    command: "npm test"
    depends_on: [build]

  - name: eval-agent
    type: eval
    depends_on: [unit-tests]
    config:
      scenario_name: "code-review-task"
      min_pass_rate: 0.9
      fail_on_regression: true
      baseline_count: 10
      regression_threshold: 0.05  # 5% max regression

  - name: quality-check
    type: gate
    depends_on: [eval-agent]
    config:
      trust_above: 80
      cost_below: 2.00

  - name: deploy
    type: command
    command: "npm run deploy"
    depends_on: [quality-check]
```

## Conditional Branching

Use `on_success` and `on_failure` conditions to branch based on gate results:

```yaml
stages:
  - name: eval-agent
    type: eval
    config:
      scenario_name: "main-workflow"
      min_pass_rate: 0.8

  - name: deploy-prod
    type: command
    command: "deploy --env=production"
    depends_on: [eval-agent]
    condition:
      on_success: true  # Only runs if eval passed

  - name: notify-failure
    type: command
    command: "slack-notify --channel=alerts"
    depends_on: [eval-agent]
    condition:
      on_failure: true  # Only runs if eval failed
```

## Gate Results

When a gate blocks a pipeline, the `gate_result` field on the stage provides details:

```json
{
  "passed": false,
  "failed_checks": [
    {
      "type": "min_pass_rate",
      "expected": "0.80",
      "actual": "0.65",
      "passed": false,
      "message": "Pass rate 65.0% below required 80.0%"
    },
    {
      "type": "regression",
      "expected": ">= 85.0% (baseline)",
      "actual": "65.0%",
      "passed": false,
      "message": "Score regressed 23.5% vs baseline (85.0% → 65.0%)"
    }
  ],
  "passed_checks": [],
  "message": "Eval gate failed: 2 check(s) failed",
  "score": 0.65,
  "baseline_score": 0.85,
  "is_regression": true,
  "regression_pct": 23.5
}
```

## Best Practices

### 1. Start with Lenient Thresholds

Begin with conservative thresholds and tighten as your agent improves:

```yaml
# Week 1: Establish baseline
min_pass_rate: 0.5
regression_threshold: 0.2  # 20%

# Week 4: Tighten as agent stabilizes
min_pass_rate: 0.8
regression_threshold: 0.1  # 10%

# Production: Strict enforcement
min_pass_rate: 0.95
regression_threshold: 0.05  # 5%
```

### 2. Use Suites for Comprehensive Testing

Run multiple scenarios as a suite for broader coverage:

```yaml
- name: full-eval
  type: eval
  config:
    suite_name: "agent-capabilities"
    min_pass_rate: 0.85
```

### 3. Set Cost Budgets

Prevent runaway costs with budget gates:

```yaml
- name: cost-check
  type: gate
  config:
    cost_below: 5.00  # $5 max per pipeline run
```

### 4. Combine with Trust Scoring

Use trust gates to ensure agent behavior stays safe:

```yaml
- name: trust-gate
  type: gate
  depends_on: [eval-agent]
  config:
    trust_above: 75  # Block if risky behavior detected
```

## Recipe: "Don't Merge Until Agent Tests Pass"

Block PRs from merging until the agent passes its own test suite:

```yaml
# .caged/pipelines/pr-check.yaml
name: pr-agent-check
version: 1

stages:
  - name: checkout
    type: command
    command: "git checkout $CAGED_PR_BRANCH"

  - name: run-agent-tests
    type: eval
    depends_on: [checkout]
    config:
      suite_name: "pr-validation-suite"
      min_pass_rate: 1.0        # All tests must pass
      fail_on_regression: true
      regression_threshold: 0   # No regression allowed
      max_cost_usd: 1.00

  - name: update-pr-status
    type: command
    depends_on: [run-agent-tests]
    command: |
      gh pr review $CAGED_PR_NUMBER --approve \
        --body "✅ Agent tests passed (score: $(cat eval_result.json | jq .score))"
    condition:
      on_success: true

  - name: block-pr
    type: command
    depends_on: [run-agent-tests]
    command: |
      gh pr review $CAGED_PR_NUMBER --request-changes \
        --body "❌ Agent tests failed. See pipeline for details."
    condition:
      on_failure: true
```

## API Reference

### Get Stage Gate Result

```bash
GET /v1/pipelines/runs/{run_id}/stages/{stage_id}
```

Response includes `gate_result` for eval and gate stages.

### List Pipeline Runs with Gate Failures

```bash
GET /v1/pipelines/{pipeline_id}/runs?status=failed&has_gate_failure=true
```

## Dashboard

The pipeline dashboard shows:
- Which gate blocked each run
- Pass/fail checks with details
- Regression trends over time
- Cost accumulation per stage

Navigate to **Pipelines → [Your Pipeline] → Runs** to see gate results inline with each stage.