# Computer Controls

Capture the screen of a macOS Devbox, click and type on it, and watch it over VNC.

Use display controls to capture screenshots of a Devbox, click anywhere on its screen, and type into the focused app. Together they let an agent see what an app shows and act on it, for example to check that a page renders or to click through a flow that has no command-line equivalent.

> [!NOTE]
> 
> Display controls are available on macOS Devboxes only. See [Create a Devbox](/content/docs/devbox/lifecycle.md#create-a-devbox/index.html).

## Take screenshots

Capture the full screen as a PNG. The screenshot is 2560 by 1600 pixels.

**CLI**

```bash {6} theme={null}
# Open a page, and wait for it to load
devbox exec my-mac-devbox -- open -a Safari https://namespace.so
sleep 2

# Capture the screen and save it as a PNG
devbox desktop screenshot my-mac-devbox --file screen.png
```

The CLI prints the screenshot size, in the same pixels that `devbox desktop click` uses. Pass `--file -` to write the PNG to stdout instead.

**TS SDK**

```typescript {12} theme={null}
import { writeFile } from "node:fs/promises";
import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Open a page, and wait for it to load
await devbox.exec(["open", "-a", "Safari", "https://namespace.so"]);
await new Promise((resolve) => setTimeout(resolve, 2_000));

// Capture the screen and save it as a PNG
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("screen.png", screenshot.png);

client.close();
```

A display call starts a stopped Devbox and keeps its connection open for later screenshots and clicks. See [`display.screenshot()`](/content/docs/reference/typescript-sdk/devbox/display#display-screenshot/index.html) for its options and return value.

The result is a full-screen capture, ready to pass to your model:

## Click the mouse

Click at a point on the screen. Coordinates are in screenshot pixels, with `(0, 0)` at the top-left corner, so a point read from a screenshot can be passed as is.

**CLI**

```bash {2} theme={null}
# Left click on the Contact Us button
devbox desktop click my-mac-devbox --x 2200 --y 280

# Wait for the page to load and capture a screenshot
sleep 2
devbox desktop screenshot my-mac-devbox --file contact-us.png
```

Pass `--button right` or `--button middle` to click another mouse button.

**TS SDK**

```typescript {7} theme={null}
import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Left click on the Contact Us button
await devbox.display.click(2200, 280);

// Wait for the page to load and capture a screenshot
await new Promise((resolve) => setTimeout(resolve, 2_000));
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("contact-us.png", screenshot.png);

client.close();
```

See [`display.click()`](/content/docs/reference/typescript-sdk/devbox/display#display-click/index.html) for its options, including the mouse button.

A click returns as soon as it is sent, before the app reacts. Wait briefly, then take a new screenshot to confirm the result before choosing the next action.

## Use the keyboard

Type text into the app that has focus, and press keys and shortcuts. Click a text field first, or use a shortcut, to choose where the text goes.

**CLI**

```bash {2,5-6, 9} theme={null}
# Focus the Safari address bar
devbox desktop press my-mac-devbox --key cmd+l

# Type a URL and press Enter
devbox desktop type my-mac-devbox --text "namespace.so/docs"
devbox desktop press my-mac-devbox --key enter

# Press several keys in order
devbox desktop press my-devbox --key down --key down --key enter
```

* [`devbox desktop type`](/content/docs/reference/devbox-cli/desktop#devbox-desktop-type/index.html) types text into the focused app.
* [`devbox desktop press`](/content/docs/reference/devbox-cli/desktop#devbox-desktop-press/index.html) presses keys and shortcuts, such as `enter` or `cmd+l`.

## Agent loop

Screenshots and clicks combine into a loop that lets an agent work on an app until a task is done:

1. Start the app with a command, for example `open -a Safari https://namespace.so`.
2. Take a screenshot to capture what the screen shows.
3. Send the screenshot and the task to a vision model, and ask it for the next action.
4. Run the action the model returns, such as a click, typing, or a key press, then wait for the app to react.
5. Repeat from step 2 until the model reports that the task is done.

The loops below follow that pattern. `decide_next_action` in the CLI script and `decideNextAction()` in the TS SDK stand for your model call, for example to the Claude or OpenAI computer use tools. It receives the screenshot and returns the next action: a click, text to type or a key to press (CLI only), a URL to open, or `done` with the task result.

**CLI**

```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail

devbox=my-mac-devbox
task="List Devbox docs pages from the sidenav. Expand collapsed sections where needed."

# Your model call: reads the task and a screenshot, prints the next action as JSON,
# such as {"type":"click","x":120,"y":340}, {"type":"type","text":"docs"},
# {"type":"press","key":"enter"}, or {"type":"done","result":"..."}
decide_next_action() {
  ./decide-next-action "$1" "$2"
}

# 1. Start the app
devbox exec "$devbox" -- open -a Safari https://namespace.so/docs
sleep 2

while true; do
  # 2. Capture the current screen
devbox desktop screenshot "$devbox" --file screen.png

# 3. Ask the model for the next action
action=$(decide_next_action "$task" screen.png)
type=$(jq -r .type <<<"$action")

# 4. Print the result and stop the loop when done
  if [ "$type" = done ]; then
    jq -r .result <<<"$action"
    break
  fi

# 5. Run the action, then give the app time to react
  case "$type" in
    click) devbox desktop click "$devbox" --x "$(jq -r .x <<<"$action")" --y "$(jq -r .y <<<"$action")" ;; 
    type) devbox desktop type "$devbox" --text "$(jq -r .text <<<"$action")" ;;
    press) devbox desktop press "$devbox" --key "$(jq -r .key <<<"$action")" ;; 
    open) devbox exec "$devbox" -- open "$(jq -r .target <<<"$action")" ;;
  esac
  sleep 2
done
```

**TS SDK**

```typescript theme={null}
import { createDevboxClient } from "@namespacelabs/sdk/devbox";

type Action =
  | { type: "click"; x: number; y: number; button?: "left" | "middle" | "right" }
  | { type: "open"; target: string }
  | { type: "done"; result: string };

// Your model call: sends the task and screenshot, returns the next action
declare function decideNextAction(task: string, screenshot: Uint8Array): Promise<Action>;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

const task = "List Devbox docs pages from the sidenav. Expand collapsed sections where needed."

// 1. Start the app
await devbox.exec(["open", "-a", "Safari", "https://namespace.so/docs"]);
await sleep(2_000);

while (true) {
  // 2. Capture the current screen
  const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });

// 3. Ask the model for the next action
  const action = await decideNextAction(task, screenshot.png);

// 4. Print the result and stop the loop when done
  if (action.type === "done") {
    console.log(action.result);
    break;
  }

// 5. Run the action, then give the app time to react
  switch (action.type) {
    case "click":
      await devbox.display.click(action.x, action.y, { button: action.button });
      break;
    case "open":
      await devbox.exec(["open", action.target]);
      break;
  }
  await sleep(2_000);
}

client.close();
```

Tell the model the screen size, 2560 by 1600 pixels, so that the coordinates it returns match the screenshot.

The `open` action shows how to mix commands into the loop. For anything that does not need the screen, such as launching apps or opening URLs, run a command with [`devbox exec`](/content/docs/devbox/exec/index.html) or `devbox.exec()`. Commands are usually faster and more reliable than working through the screen.

## Watch the desktop

Open the desktop in a VNC client on your machine to watch an agent work, unblock it when it is stuck on a dialog, or take over with your own mouse and keyboard. See [Remote Desktop](/content/docs/devbox/remote-desktop.md).

**CLI**

```bash theme={null}
devbox desktop connect my-mac-devbox
```

You and an agent share one pointer and keyboard, so input from both at the same time collides. Ask the agent to pause before you take over, and have it take a new screenshot before it acts again.