Guides
View as Markdown Open in ChatGPT Open in Claude

Desktop Sandboxes

Desktop sandboxes provide a complete virtual desktop environment for AI agents that need visual interaction — like Claude's Computer Use, OpenAI's Operator, or custom screen-grounded agents.

Unlike browser-only automation, desktop sandboxes give agents a full Linux GUI with window manager, allowing them to interact with any application.

Quick Start

# Create a desktop sandbox
caged run --template desktop

# Or use aliases
caged run --template gui
caged run --template computer

import caged

# Create desktop sandbox
sandbox = caged.create(template="desktop")

# Take a screenshot
screenshot = sandbox.mcp.call("screen_screenshot", {})
print(f"Screen size: {screenshot['width']}x{screenshot['height']}")

# Click at coordinates
sandbox.mcp.call("screen_click", {"x": 640, "y": 400})

# Type text
sandbox.mcp.call("screen_type", {"text": "Hello, world!"})

# Press Enter
sandbox.mcp.call("screen_key", {"key": "Return"})

What's Included

The desktop template includes:

Component Description
Xvfb Virtual framebuffer (1280×800, 24-bit)
Openbox Lightweight window manager
x11vnc VNC server for remote viewing
Chromium Web browser
xdotool Input automation
scrot Screenshots

Screen Tools

screen_screenshot

Capture the current screen state.

// Request
{"name": "screen_screenshot", "arguments": {}}

// Response
{
  "image_base64": "iVBORw0KGgo...",
  "width": 1280,
  "height": 800,
  "format": "png",
  "captured_at": "2026-01-15T10:30:00Z"
}

screen_click

Click at specific coordinates.

{
  "name": "screen_click",
  "arguments": {
    "x": 640,
    "y": 400,
    "button": "left",   // "left", "right", "middle"
    "double": false     // true for double-click
  }
}

screen_type

Type text using the keyboard.

{
  "name": "screen_type",
  "arguments": {
    "text": "Hello, world!",
    "delay": 50          // milliseconds between keystrokes
  }
}

screen_key

Press keyboard keys including special keys and combinations.

{
  "name": "screen_key",
  "arguments": {
    "key": "Return"      // or "Tab", "Escape", "ctrl+c", "alt+Tab", "F1", etc.
  }
}

Common keys:

  • Return — Enter
  • Tab, Escape, BackSpace, Delete
  • Up, Down, Left, Right — Arrow keys
  • ctrl+c, ctrl+v, ctrl+z — Shortcuts
  • alt+Tab, alt+F4 — Window switching
  • super — Windows/Meta key
  • F1 through F12

screen_scroll

Scroll at specific coordinates.

{
  "name": "screen_scroll",
  "arguments": {
    "x": 640,
    "y": 400,
    "direction": "down",  // "up" or "down"
    "amount": 3           // scroll increments
  }
}

screen_drag

Drag from one point to another.

{
  "name": "screen_drag",
  "arguments": {
    "start_x": 100,
    "start_y": 100,
    "end_x": 300,
    "end_y": 200
  }
}

screen_launch

Launch an application.

{
  "name": "screen_launch",
  "arguments": {
    "command": "chromium-browser https://example.com",
    "wait": true   // wait for app to start
  }
}

screen_info

Get current screen state.

// Response
{
  "width": 1280,
  "height": 800,
  "mouse_x": 640,
  "mouse_y": 400,
  "active_window": "Chromium"
}

screen_wait

Wait for a duration (useful for animations, page loads).

{
  "name": "screen_wait",
  "arguments": {
    "seconds": 2    // max 30 seconds
  }
}

WebSocket Streaming

For real-time screen interaction, connect via WebSocket:

const ws = new WebSocket(
  `wss://api.caged.dev/v1/sandboxes/${sandboxId}/screen`,
  {
    headers: { Authorization: `Bearer ${apiKey}` }
  }
);

// Request screenshot
ws.send(JSON.stringify({ type: "screenshot" }));

// Receive screenshot
ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "screenshot") {
    const img = document.getElementById("screen");
    img.src = `data:image/png;base64,${msg.image_base64}`;
  }
};

// Send click
ws.send(JSON.stringify({ type: "click", x: 640, y: 400 }));

VNC Access

For direct VNC access (debugging, manual intervention):

# Get VNC info
curl https://api.caged.dev/v1/sandboxes/sbx_abc123/screen/vnc \
  -H "Authorization: Bearer $CAGED_API_KEY"

# Response
{
  "sandbox_id": "sbx_abc123",
  "vnc_port": 5900,
  "status": "available"
}

Use port forwarding to connect with a VNC client:

# Enable port forwarding
caged ports enable sbx_abc123 5900

# Connect with VNC client to the forwarded URL

Claude Computer Use Integration

Desktop sandboxes work seamlessly with Claude's Computer Use capability:

import anthropic
import caged

# Create sandbox
sandbox = caged.create(template="desktop")

# Initialize Claude with computer use
client = anthropic.Anthropic()

def handle_tool_call(tool_name, tool_input):
    if tool_name == "computer":
        action = tool_input["action"]
        if action == "screenshot":
            return sandbox.mcp.call("screen_screenshot", {})
        elif action == "mouse_move":
            x, y = tool_input["coordinate"]
            return sandbox.mcp.call("screen_click", {"x": x, "y": y})
        elif action == "type":
            return sandbox.mcp.call("screen_type", {"text": tool_input["text"]})
        elif action == "key":
            return sandbox.mcp.call("screen_key", {"key": tool_input["key"]})
        # ... handle other actions
    # ... handle other tools

# Run agent loop
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=4096,
    tools=[{
        "type": "computer_20241022",
        "name": "computer",
        "display_width": 1280,
        "display_height": 800
    }],
    messages=[{"role": "user", "content": "Open Chrome and search for 'Caged AI'"}]
)

# Process tool calls
for block in response.content:
    if block.type == "tool_use":
        result = handle_tool_call(block.name, block.input)
        # Continue conversation with result...

Session Replay

All desktop actions are recorded in session replay:

  • Screenshots captured before/after actions
  • Click coordinates and button states
  • Typed text (sensitive data redacted)
  • Key presses and combinations
  • Application launches

View replay in the dashboard or via API:

GET /v1/sessions/{sessionId}/replay

Trust Scoring

Desktop actions affect trust scores:

Action Impact Reason
Screenshots +1 Agent observing before acting
Browser launch +1 Normal application use
Typing credentials -10 Sensitive text detected
Suspicious commands -15 Potentially malicious

Resource Requirements

Desktop sandboxes require more resources:

Resource Requirement
Memory 2GB minimum (4GB recommended)
vCPU 2 cores
Disk 8GB

Specify in .caged.yaml:

template: desktop
resources:
  memory: 4096  # MB
  cpu: 2
  disk: 8192    # MB

Best Practices

Always take a screenshot before interacting to understand the current state

Use screen_wait after launching apps or navigating to let the UI settle

Set typing delays (50-100ms) to avoid missing keystrokes

Retry actions if they don't produce expected results

Limitations

  • Resolution: Fixed at 1280×800 (configurable via environment variables)
  • GPU: No hardware acceleration (software rendering only)
  • Audio: Not supported
  • Clipboard: Use screen_type and screen_key for copy/paste
Was this page helpful?