> ## Documentation Index
> Fetch the complete documentation index at: https://docs.caged.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Eval & Testing

> Define test scenarios for agents with assertions, flakiness detection, and regression testing

# Agent Eval & Testing (Cage Eval)

Cage Eval lets you define reproducible test scenarios for AI agents. Run agents against assertions, detect flaky behavior, and catch regressions before shipping.

## Concepts

* **Scenario** — A single test case: a prompt, a sandbox template, and assertions to verify the output
* **Suite** — A group of related scenarios for batch execution
* **Run** — A single execution of a scenario, producing assertion results and a score
* **Assertion** — A verification check (file exists, output contains, cost below, etc.)

## YAML File Format

### Suite File (`.caged-eval.yaml`)

```yaml theme={null}
name: "API Agent Tests"
description: "Verify the agent can build REST APIs"
tags: [api, backend]
defaults:
  template: node-20
  budget: 0.50
  timeout: 5m
  model: claude-sonnet-4-20250514
scenarios:
  - name: "Create Express server"
    prompt: "Create a basic Express.js server with a /health endpoint"
    assertions:
      - { id: server, type: file_exists, path: "server.js" }
      - { id: health, type: file_contains, path: "server.js", contains: "/health" }
      - { id: cost, type: cost_below, max_cost: 0.25 }

  - name: "Add unit tests"
    prompt: "Add Jest tests for the server"
    assertions:
      - { id: tests, type: file_exists, path: "test/server.test.js" }
      - { id: passes, type: exit_code, command: "npm test", exit_code: 0 }
```

### Single Scenario File (`*.scenario.yaml`)

```yaml theme={null}
name: "Hello World Python"
template: python-3.12
prompt: "Create a hello world Python script"
budget: 0.10
assertions:
  - { id: main, type: file_exists, path: "main.py" }
  - { id: output, type: output_contains, command: "python main.py", contains: "Hello" }
```

## Assertion Types

| Type              | Description                    | Key Fields             |
| ----------------- | ------------------------------ | ---------------------- |
| `file_exists`     | File exists at path            | `path`                 |
| `file_contains`   | File contains substring        | `path`, `contains`     |
| `file_matches`    | File content matches regex     | `path`, `pattern`      |
| `exit_code`       | Command exits with code        | `command`, `exit_code` |
| `output_contains` | Command output contains string | `command`, `contains`  |
| `output_matches`  | Command output matches regex   | `command`, `pattern`   |
| `no_errors`       | No error events in session     | —                      |
| `cost_below`      | Total cost below threshold     | `max_cost`             |
| `trust_above`     | Trust score above threshold    | `min_score`            |
| `custom`          | Custom script returns 0        | `command`              |

## CLI Commands

```bash theme={null}
# Discover scenario files in current directory
caged eval discover

# Load a suite file (creates suite + scenarios via API)
caged eval load .caged-eval.yaml

# List all scenarios
caged eval list

# Run a specific scenario
caged eval run <scenario-id>

# Run flakiness test (10 iterations)
caged eval flakiness <scenario-id> --iterations 10

# Detect regression against last 5 runs
caged eval regression <scenario-id>

# View run history
caged eval runs <scenario-id>
```

## Flakiness Detection

Run a scenario N times and measure consistency:

```bash theme={null}
caged eval flakiness <id> --iterations 10
```

Returns:

* Pass rate (e.g. 8/10 = 80%)
* Per-assertion flakiness breakdown
* Verdict: stable, flaky, or broken

## Regression Detection

Compare current run against baseline (last N runs):

```bash theme={null}
caged eval regression <id>
```

Detects:

* Score drops (>10% threshold)
* New failing assertions
* Cost increases (>20% threshold)

## CI/CD Integration

Add to your CI pipeline:

```yaml theme={null}
# GitHub Actions
- name: Run agent evals
  run: |
    caged eval load .caged-eval.yaml
    caged eval run --suite $SUITE_ID --fail-on-regression
```

## API

```bash theme={null}
# Create scenario
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -d '{"name":"test","template":"node-20","prompt":"...","assertions":[...]}' \
  https://api.caged.dev/v1/eval/scenarios

# Run scenario
curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://api.caged.dev/v1/eval/scenarios/{id}/run

# Get run results
curl -H "Authorization: Bearer $TOKEN" \
  https://api.caged.dev/v1/eval/runs/{run-id}
```