--- title: "Desktop Sandboxes" description: "Run screen-grounded AI agents with full virtual desktop environments" --- # Desktop Sandboxes Desktop sandboxes provide a complete virtual desktop environment for AI agents that need visual interaction — like Claude's Computer Use, OpenAI's Operator, or custom screen-grounded agents. Unlike browser-only automation, desktop sandboxes give agents a full Linux GUI with window manager, allowing them to interact with any application. ## Quick Start ```bash # Create a desktop sandbox caged run --template desktop # Or use aliases caged run --template gui caged run --template computer ``` ```python import caged # Create desktop sandbox sandbox = caged.create(template="desktop") # Take a screenshot screenshot = sandbox.mcp.call("screen_screenshot", {}) print(f"Screen size: {screenshot['width']}x{screenshot['height']}") # Click at coordinates sandbox.mcp.call("screen_click", {"x": 640, "y": 400}) # Type text sandbox.mcp.call("screen_type", {"text": "Hello, world!"}) # Press Enter sandbox.mcp.call("screen_key", {"key": "Return"}) ``` ## What's Included The desktop template includes: | Component | Description | |-----------|-------------| | Xvfb | Virtual framebuffer (1280×800, 24-bit) | | Openbox | Lightweight window manager | | x11vnc | VNC server for remote viewing | | Chromium | Web browser | | xdotool | Input automation | | scrot | Screenshots | ## Screen Tools ### screen_screenshot Capture the current screen state. ```json // Request {"name": "screen_screenshot", "arguments": {}} // Response { "image_base64": "iVBORw0KGgo...", "width": 1280, "height": 800, "format": "png", "captured_at": "2026-01-15T10:30:00Z" } ``` ### screen_click Click at specific coordinates. ```json { "name": "screen_click", "arguments": { "x": 640, "y": 400, "button": "left", // "left", "right", "middle" "double": false // true for double-click } } ``` ### screen_type Type text using the keyboard. ```json { "name": "screen_type", "arguments": { "text": "Hello, world!", "delay": 50 // milliseconds between keystrokes } } ``` ### screen_key Press keyboard keys including special keys and combinations. ```json { "name": "screen_key", "arguments": { "key": "Return" // or "Tab", "Escape", "ctrl+c", "alt+Tab", "F1", etc. } } ``` Common keys: - `Return` — Enter - `Tab`, `Escape`, `BackSpace`, `Delete` - `Up`, `Down`, `Left`, `Right` — Arrow keys - `ctrl+c`, `ctrl+v`, `ctrl+z` — Shortcuts - `alt+Tab`, `alt+F4` — Window switching - `super` — Windows/Meta key - `F1` through `F12` ### screen_scroll Scroll at specific coordinates. ```json { "name": "screen_scroll", "arguments": { "x": 640, "y": 400, "direction": "down", // "up" or "down" "amount": 3 // scroll increments } } ``` ### screen_drag Drag from one point to another. ```json { "name": "screen_drag", "arguments": { "start_x": 100, "start_y": 100, "end_x": 300, "end_y": 200 } } ``` ### screen_launch Launch an application. ```json { "name": "screen_launch", "arguments": { "command": "chromium-browser https://example.com", "wait": true // wait for app to start } } ``` ### screen_info Get current screen state. ```json // Response { "width": 1280, "height": 800, "mouse_x": 640, "mouse_y": 400, "active_window": "Chromium" } ``` ### screen_wait Wait for a duration (useful for animations, page loads). ```json { "name": "screen_wait", "arguments": { "seconds": 2 // max 30 seconds } } ``` ## WebSocket Streaming For real-time screen interaction, connect via WebSocket: ```javascript const ws = new WebSocket( `wss://api.caged.dev/v1/sandboxes/${sandboxId}/screen`, { headers: { Authorization: `Bearer ${apiKey}` } } ); // Request screenshot ws.send(JSON.stringify({ type: "screenshot" })); // Receive screenshot ws.onmessage = (event) => { const msg = JSON.parse(event.data); if (msg.type === "screenshot") { const img = document.getElementById("screen"); img.src = `data:image/png;base64,${msg.image_base64}`; } }; // Send click ws.send(JSON.stringify({ type: "click", x: 640, y: 400 })); ``` ## VNC Access For direct VNC access (debugging, manual intervention): ```bash # Get VNC info curl https://api.caged.dev/v1/sandboxes/sbx_abc123/screen/vnc \ -H "Authorization: Bearer $CAGED_API_KEY" # Response { "sandbox_id": "sbx_abc123", "vnc_port": 5900, "status": "available" } ``` Use port forwarding to connect with a VNC client: ```bash # Enable port forwarding caged ports enable sbx_abc123 5900 # Connect with VNC client to the forwarded URL ``` ## Claude Computer Use Integration Desktop sandboxes work seamlessly with Claude's Computer Use capability: ```python import anthropic import caged # Create sandbox sandbox = caged.create(template="desktop") # Initialize Claude with computer use client = anthropic.Anthropic() def handle_tool_call(tool_name, tool_input): if tool_name == "computer": action = tool_input["action"] if action == "screenshot": return sandbox.mcp.call("screen_screenshot", {}) elif action == "mouse_move": x, y = tool_input["coordinate"] return sandbox.mcp.call("screen_click", {"x": x, "y": y}) elif action == "type": return sandbox.mcp.call("screen_type", {"text": tool_input["text"]}) elif action == "key": return sandbox.mcp.call("screen_key", {"key": tool_input["key"]}) # ... handle other actions # ... handle other tools # Run agent loop response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=4096, tools=[{ "type": "computer_20241022", "name": "computer", "display_width": 1280, "display_height": 800 }], messages=[{"role": "user", "content": "Open Chrome and search for 'Caged AI'"}] ) # Process tool calls for block in response.content: if block.type == "tool_use": result = handle_tool_call(block.name, block.input) # Continue conversation with result... ``` ## Session Replay All desktop actions are recorded in session replay: - Screenshots captured before/after actions - Click coordinates and button states - Typed text (sensitive data redacted) - Key presses and combinations - Application launches View replay in the dashboard or via API: ```bash GET /v1/sessions/{sessionId}/replay ``` ## Trust Scoring Desktop actions affect trust scores: | Action | Impact | Reason | |--------|--------|--------| | Screenshots | +1 | Agent observing before acting | | Browser launch | +1 | Normal application use | | Typing credentials | -10 | Sensitive text detected | | Suspicious commands | -15 | Potentially malicious | ## Resource Requirements Desktop sandboxes require more resources: | Resource | Requirement | |----------|-------------| | Memory | 2GB minimum (4GB recommended) | | vCPU | 2 cores | | Disk | 8GB | Specify in `.caged.yaml`: ```yaml template: desktop resources: memory: 4096 # MB cpu: 2 disk: 8192 # MB ``` ## Best Practices Always take a screenshot before interacting to understand the current state Use `screen_wait` after launching apps or navigating to let the UI settle Set typing delays (50-100ms) to avoid missing keystrokes Retry actions if they don't produce expected results ## Limitations - **Resolution**: Fixed at 1280×800 (configurable via environment variables) - **GPU**: No hardware acceleration (software rendering only) - **Audio**: Not supported - **Clipboard**: Use `screen_type` and `screen_key` for copy/paste ## Related - [Computer Use Agent Recipe](/recipes/computer-use) — Full example with Claude Computer Use - [Session Replay](/guides/replay) — Viewing agent actions - [Trust Scoring](/guides/trust-scoring) — Understanding trust scores - [Policies](/guides/policies) — Configuring security policies