--- title: Pipeline State Store description: "Share data between pipeline stages without polluting git history." --- # Pipeline State Store The pipeline state store is a key/value store scoped to each pipeline run. Stages can read and write state entries to share data — analysis results, computed artifacts, configuration decisions — without committing files to git or using external storage. ## Why State Store? Before state store, multi-stage pipelines had awkward options for sharing data: | Pattern | Problem | |---------|---------| | Commit to git branch | Pollutes history, slow clone times, merge conflicts | | External S3/Redis | Extra infrastructure, credentials, cleanup | | Environment variables | Size limits, can't pass structured data | | File system | Lost when sandbox terminates | The state store provides: - **Structured data** — JSON, strings, file references, patches - **Automatic cleanup** — TTL-based expiration, no orphaned artifacts - **Access control** — Scoped to the run, no cross-pipeline leakage - **Observable** — State visible in dashboard, API, CLI - **Size limits** — Prevents unbounded growth (10 MB per run) ## Core Concepts ### State Entry A key/value pair with metadata: ```json { "key": "analysis_results", "value": "{\"files\": [\"src/main.ts\"], \"score\": 85}", "type": "json", "mime_type": "application/json", "size_bytes": 42, "created_by": "analyze", "created_at": "2026-08-02T10:05:30Z", "expires_at": "2026-08-09T10:05:30Z" } ``` ### Value Types | Type | Description | Use Case | |------|-------------|----------| | `string` | Plain text | Status flags, IDs, short messages | | `json` | JSON object/array | Analysis results, configurations, lists | | `file` | File path reference | Build artifacts, generated files | | `patch` | Git diff format | Code changes for review | | `artifact` | Binary blob reference | Compiled outputs, images | ### TTL (Time-to-Live) State entries expire automatically: - **Default TTL**: 7 days - **Maximum TTL**: 30 days - **Custom TTL**: Set per-entry when writing Expired entries are cleaned up automatically — no manual garbage collection needed. ## Usage Patterns ### Writing State from Stages Agent stages can write state using the `caged state` command available inside sandboxes: ```yaml stages: - name: analyze type: agent agent: claude-code prompt: | Analyze the codebase for refactoring opportunities. When done, save your findings: caged state set analysis '{"files": [...], "priority": "high"}' ``` Command stages can also write state: ```yaml stages: - name: compute-hash type: command command: | HASH=$(sha256sum ./dist/app.js | cut -d' ' -f1) caged state set build_hash "$HASH" ``` ### Reading State in Subsequent Stages Later stages read state to continue the work: ```yaml stages: - name: refactor type: agent agent: claude-code prompt: | Read the analysis results and refactor the identified files. First, get the analysis: caged state get analysis Then refactor each file listed in the results. depends_on: [analyze] ``` ### Multi-Agent Handoff The most powerful pattern — multiple specialized agents collaborate through state: ```yaml pipelines: - name: ai-code-review stages: # Agent 1: Implements the feature - name: implement type: agent agent: claude-code prompt: | Implement: $FEATURE_SPEC When complete, summarize what you did: caged state set implementation '{ "files_changed": ["src/feature.ts", "src/feature.test.ts"], "approach": "Used factory pattern for extensibility", "test_coverage": 87 }' --type json budget: 15.00 # Agent 2: Security review (reads implementation context) - name: security-review type: agent agent: claude-code prompt: | Review the implementation for security issues. Get context: caged state get implementation Save your findings: caged state set security_findings '{ "issues": [...], "severity": "low", "approved": true }' --type json depends_on: [implement] budget: 5.00 # Agent 3: Performance review (parallel with security) - name: perf-review type: agent agent: aider prompt: | Review for performance issues. caged state get implementation caged state set perf_findings '{"issues": [...]}' --type json depends_on: [implement] budget: 5.00 # Final stage: Compile all findings - name: compile-report type: command command: | impl=$(caged state get implementation) security=$(caged state get security_findings) perf=$(caged state get perf_findings) echo "{ \"implementation\": $impl, \"security\": $security, \"performance\": $perf }" | jq '.' > report.json caged state set final_report -f report.json --type json depends_on: [security-review, perf-review] ``` DAG visualization: ``` implement / \ security-review perf-review \ / compile-report ``` Each agent sees only what it needs, writes its findings, and the final stage aggregates everything. ### Incremental Processing For long-running migrations, track progress in state: ```yaml stages: - name: scan type: command command: | # Find all files to migrate find src -name "*.js" > /tmp/files.txt caged state set remaining "$(cat /tmp/files.txt | jq -R . | jq -s .)" --type json caged state set completed "[]" --type json - name: migrate-batch type: agent agent: claude-code prompt: | Get the migration state: remaining=$(caged state get remaining) completed=$(caged state get completed) Process 10 files from 'remaining'. Move processed files to 'completed'. Update state when done. If all files are migrated, write: caged state set migration_complete "true" depends_on: [scan] timeout: 30m - name: continue-check type: gate depends_on: [migrate-batch] gate: # Continue if migration not complete if: "state['migration_complete'] != 'true'" ``` ### Human Approval with Context Pass context to human reviewers through state: ```yaml stages: - name: implement type: agent agent: claude-code prompt: | Implement the feature and save a summary for human review: caged state set review_summary '{ "what_changed": "Added user preferences API", "files": 5, "test_coverage": 92, "breaking_changes": false }' --type json - name: approve type: await_approval depends_on: [implement] approval: message: | Agent completed implementation. Review summary available at: caged pipeline state get $PIPELINE_ID $RUN_ID review_summary channels: [slack, dashboard] ``` ## CLI Usage ### Outside the Sandbox Manage state from your local machine: ```bash # List all state entries caged pipeline state list pipe-abc123 run-xyz789 # Get a specific entry caged pipeline state get pipe-abc123 run-xyz789 analysis_results # Set a value caged pipeline state set pipe-abc123 run-xyz789 config '{"debug": true}' --type json # Set from file caged pipeline state set pipe-abc123 run-xyz789 report -f ./report.json --type json --ttl 86400 # Delete an entry caged pipeline state delete pipe-abc123 run-xyz789 temp_data ``` ### Inside the Sandbox Within a pipeline stage, use the simpler form (run context is implicit): ```bash # These work inside pipeline sandboxes caged state set my_key "my_value" caged state get my_key caged state list ``` ## API Usage Full CRUD via REST API: ```bash # List state curl https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state \ -H "Authorization: Bearer caged_sk_..." # Get single entry curl https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \ -H "Authorization: Bearer caged_sk_..." # Set entry curl -X PUT https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \ -H "Authorization: Bearer caged_sk_..." \ -H "Content-Type: application/json" \ -d '{ "value": {"result": "success"}, "type": "json", "ttl_seconds": 86400 }' # Delete entry curl -X DELETE https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \ -H "Authorization: Bearer caged_sk_..." ``` ## MCP Tools When using the Caged MCP server, four state tools are available: | Tool | Description | |------|-------------| | `pipeline_state_list` | List all state entries for a run | | `pipeline_state_get` | Get a single state entry | | `pipeline_state_set` | Create or update a state entry | | `pipeline_state_delete` | Delete a state entry | See [MCP Server Reference](/open-source/mcp-server) for full tool documentation. ## Limits | Limit | Value | Rationale | |-------|-------|-----------| | Max key length | 256 characters | Prevent key abuse | | Max value size | 1 MB | Keep state fast, use objstore for large files | | Max entries per run | 100 | Prevent key explosion | | Max total size per run | 10 MB | Bound memory/storage | | Default TTL | 7 days | Auto-cleanup | | Max TTL | 30 days | Prevent permanent storage | For artifacts larger than 1 MB, store them in object storage and write the URL/path to state. ## Best Practices 1. **Use descriptive keys** — `security_review_findings` not `data1` 2. **Set appropriate TTL** — Short TTL for temp data, longer for audit trails 3. **Use JSON for structured data** — Easier to parse in subsequent stages 4. **Include metadata** — Timestamp, creator, version in your JSON values 5. **Don't store secrets** — State is visible in dashboard/API 6. **Clean up when done** — Delete temp entries to stay under limits ## Related - [Pipelines Guide](/guides/pipelines) — Pipeline concepts and DAG execution - [Config-as-Code Pipelines](/recipes/config-pipelines) — Declarative `.caged.yaml` pipelines - [Pipelines API Reference](/api-reference/pipelines) — Full endpoint documentation - [CLI Commands](/cli/commands#caged-pipeline-state) — CLI reference