--- title: Computer Use Agent description: "Build a screen-grounded AI agent that can interact with a full desktop environment." --- # Computer Use Agent This recipe creates a full virtual desktop sandbox and connects it to Claude's Computer Use capability — letting the agent see the screen, click, type, and launch applications. ## Prerequisites - Caged CLI installed (`brew install caged-dev/tap/caged` or `curl -fsSL https://get.caged.dev | sh`) - Caged account with API key (`caged login`) - An `ANTHROPIC_API_KEY` for Claude Computer Use ## Quick Start ### CLI ```bash # Create a desktop sandbox caged up --template desktop --budget 5 # Get sandbox ID SANDBOX_ID=$(caged list --format json | jq -r '.[0].id') # Take a screenshot via API curl "https://api.caged.dev/v1/sandboxes/$SANDBOX_ID/screen/screenshot?format=base64" \ -H "Authorization: Bearer $CAGED_API_KEY" ``` ### Python SDK ```python import caged # Create desktop sandbox sandbox = caged.create(template="desktop", budget=5.0) # Take a screenshot result = sandbox.mcp.call("screen_screenshot", {}) print(f"Screen: {result['width']}x{result['height']}") # Launch browser sandbox.mcp.call("screen_launch", {"command": "chromium-browser https://example.com"}) sandbox.mcp.call("screen_wait", {"seconds": 3}) # Click on something sandbox.mcp.call("screen_click", {"x": 640, "y": 300}) # Type text sandbox.mcp.call("screen_type", {"text": "Hello from Caged!"}) sandbox.mcp.call("screen_key", {"key": "Return"}) ``` ## Full Computer Use Agent This example creates a complete agent loop that connects Claude's Computer Use to a Caged desktop sandbox: ```python import anthropic import caged import base64 import json from typing import Any def run_computer_use_agent(task: str, budget: float = 10.0): """Run a Computer Use agent in an isolated Caged sandbox.""" # Create desktop sandbox sandbox = caged.create( template="desktop", budget=budget, resources={"memory": 4096, "cpu": 2} ) print(f"Sandbox created: {sandbox.id}") # Initialize Claude client = anthropic.Anthropic() # Computer Use tool definition computer_tool = { "type": "computer_20241022", "name": "computer", "display_width_px": 1280, "display_height_px": 800, "display_number": 0 } def execute_computer_action(action: str, **kwargs) -> dict: """Execute a computer action in the sandbox.""" if action == "screenshot": result = sandbox.mcp.call("screen_screenshot", {}) return { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": result["image_base64"] } } elif action == "mouse_move": x, y = kwargs["coordinate"] sandbox.mcp.call("screen_click", {"x": x, "y": y}) return {"success": True} elif action == "left_click": sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0)}) return {"success": True} elif action == "right_click": sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0), "button": "right"}) return {"success": True} elif action == "double_click": sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0), "double": True}) return {"success": True} elif action == "type": sandbox.mcp.call("screen_type", {"text": kwargs["text"]}) return {"success": True} elif action == "key": sandbox.mcp.call("screen_key", {"key": kwargs["key"]}) return {"success": True} elif action == "scroll": direction = "down" if kwargs.get("coordinate", [0, 1])[1] > 0 else "up" info = sandbox.mcp.call("screen_info", {}) sandbox.mcp.call("screen_scroll", { "x": info.get("mouse_x", 640), "y": info.get("mouse_y", 400), "direction": direction, "amount": abs(kwargs.get("coordinate", [0, 3])[1]) }) return {"success": True} else: return {"error": f"Unknown action: {action}"} # Start agent loop messages = [{"role": "user", "content": task}] while True: response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=4096, tools=[computer_tool], messages=messages ) # Check for end conditions if response.stop_reason == "end_turn": # Extract final text response for block in response.content: if hasattr(block, "text"): print(f"\nAgent completed: {block.text}") break # Process tool calls tool_results = [] for block in response.content: if block.type == "tool_use" and block.name == "computer": action = block.input.get("action") print(f"Action: {action}") result = execute_computer_action(action, **block.input) tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": [result] if action == "screenshot" else json.dumps(result) }) # Continue conversation messages.append({"role": "assistant", "content": response.content}) messages.append({"role": "user", "content": tool_results}) # Check budget if sandbox.cost > budget * 0.9: print(f"Warning: Approaching budget limit (${sandbox.cost:.2f}/${budget})") # Cleanup sandbox.destroy() print(f"Total cost: ${sandbox.cost:.2f}") # Run the agent run_computer_use_agent( task="Open Chrome, go to news.ycombinator.com, and summarize the top 3 stories", budget=5.0 ) ``` ## Config File For reproducible setups, use `.caged.yaml`: ```yaml # .caged.yaml template: desktop resources: memory: 4096 # 4GB recommended for desktop cpu: 2 disk: 10240 # 10GB for browser cache, downloads timeout: 7200 # 2 hours max budget: 10.00 # $10 hard limit env: DISPLAY: ":99" SCREEN_WIDTH: "1280" SCREEN_HEIGHT: "800" ``` Then run: ```bash caged up ``` ## Available Screen Tools | Tool | Description | |------|-------------| | `screen_screenshot` | Capture PNG screenshot (base64) | | `screen_click` | Click at coordinates (left/right/double) | | `screen_type` | Type text with optional delay | | `screen_key` | Press keys (Return, Tab, ctrl+c, etc.) | | `screen_scroll` | Scroll up/down at coordinates | | `screen_drag` | Drag from start to end coordinates | | `screen_launch` | Launch an application | | `screen_info` | Get screen size, mouse position, active window | | `screen_wait` | Wait for animations/page loads (max 30s) | ## WebSocket Streaming For real-time screen updates, use the WebSocket endpoint: ```javascript const ws = new WebSocket( `wss://api.caged.dev/v1/sandboxes/${sandboxId}/screen`, { headers: { Authorization: `Bearer ${apiKey}` } } ); // Request continuous screenshots setInterval(() => { ws.send(JSON.stringify({ type: "screenshot" })); }, 500); // Handle responses ws.onmessage = (event) => { const msg = JSON.parse(event.data); if (msg.type === "screenshot") { updateScreenDisplay(msg.image_base64); } }; // Send interactions function click(x, y) { ws.send(JSON.stringify({ type: "click", x, y })); } ``` ## Observability All screen actions are captured in session replay: ```bash # View session replay caged sessions list caged replay sess_xxxxx # Or via API curl "https://api.caged.dev/v1/sessions/sess_xxxxx/replay" \ -H "Authorization: Bearer $CAGED_API_KEY" ``` The replay includes: - Screenshots before and after each action - Click coordinates and button states - Typed text (sensitive data redacted) - Application launches - Trust score changes ## Trust Scoring Desktop actions affect the session trust score: | Action | Score Impact | |--------|--------------| | Taking screenshots | +1 (observation before action) | | Launching browser | +1 (normal application use) | | Typing credential-like text | -10 (potential credential exposure) | | Running suspicious commands | -15 (potentially malicious) | Set a minimum trust threshold in your policy: ```yaml # Policy requiring high trust policy: trust: min_score: 70 action: pause # pause for approval if score drops ``` ## Best Practices Always take a screenshot before interacting to understand screen state Use `screen_wait` after launching apps or loading pages Desktop sandboxes use more resources — always set a budget limit For real-time control, use WebSocket instead of REST ## Limitations - **Resolution**: Default 1280×800 (configurable via env vars) - **GPU**: Software rendering only (no hardware acceleration) - **Audio**: Not supported - **Clipboard**: Use `screen_type` and `screen_key` for copy/paste - **Multiple monitors**: Single display only ## Related - [Desktop Sandboxes Guide](/guides/desktop-sandboxes) — Full reference for desktop tools - [Session Replay](/guides/replay) — Viewing agent actions - [Trust Scoring](/guides/trust-scoring) — Understanding trust scores - [Policies](/guides/policies) — Configuring security policies