Custom Computer Use Driver

A custom driver connects createComputerUse to a machine that has no built-in driver, such as a VM, a remote desktop, or a lab device.

Write a ComputerDriver, and computer use runs on it with no other change.

1. Write the driver#

A driver is an object with a screen size and a small set of methods. This example sends each method to an HTTP agent that runs on the machine.

// app/my-driver.ts
import type { ComputerDriver } from '@tanstack/ai-computer-use'

export function createMyDriver(base: string) {
  async function send(path: string, body: object) {
    const response = await fetch(`${base}${path}`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(body),
    })
    if (!response.ok) throw new Error(`${path} failed: ${response.status}`)
  }

  const driver: ComputerDriver = {
    display: { width: 1920, height: 1080 },
    environment: 'windows',
    async screenshot() {
      const response = await fetch(`${base}/screenshot`)
      if (!response.ok) throw new Error(`/screenshot failed: ${response.status}`)
      return new Uint8Array(await response.arrayBuffer())
    },
    click: (x, y, { button, count }) => send('/click', { x, y, button, count }),
    move: (x, y) => send('/move', { x, y }),
    mouseDown: (button) => send('/mouse-down', { button }),
    mouseUp: (button) => send('/mouse-up', { button }),
    scroll: (x, y, deltaX, deltaY) => send('/scroll', { x, y, deltaX, deltaY }),
    keyDown: (key) => send('/key-down', { key }),
    keyUp: (key) => send('/key-up', { key }),
    press: (keys) => send('/press', { keys }),
    type: (text) => send('/type', { text }),
  }
  return driver
}

2. Give it to createComputerUse#

import { createComputerUse } from '@tanstack/ai-computer-use'
import type { ComputerDriver } from '@tanstack/ai-computer-use'

// From app/my-driver.ts above.
declare function createMyDriver(base: string): ComputerDriver

const computerUse = createComputerUse({
  driver: createMyDriver('http://localhost:8000'),
})

Give computerUse.tools, computerUse.middleware, and computerUse.interrupt to chat(), as in the computer use setup.

3. Try it#

Start your agent on the machine, and send a task in the chat. Your agent gets the first /screenshot request, then the clicks and the key presses. The model now uses your own machine.

How it works#

Your driver does only the small steps. The core builds the rest from them:

• Drags: move, mouseDown, more move calls, then mouseUp.
• Modifier clicks and scrolls: keyDown for each modifier, the action, then keyUp in reverse order.
• hold_key and wait: the core holds the keys or waits, then continues.
• Key names: the core changes OpenAI and Anthropic key names to KeyboardEvent.key names.
• Screenshots: after a normal action, the core waits screenshotDelay ms and calls screenshot(). An OpenAI batch waits once, after the last action, then takes one screenshot. zoom and cursor_position return their own result, so the core does not call screenshot() for them.

A driver is one machine. Every thread shares it, so it serves one conversation at a time. createComputerUse never closes it. To give each thread its own machine, write a driver factory.

Advanced#

The contract#

Coordinates are in pixels, in display space. Key names are KeyboardEvent.key names, for example 'Enter', 'Control', or 'a'.

Each action also takes an optional signal, the abort signal of the tool call. It is the last argument, for example move(x, y, { signal }). click gets it in its options: click(x, y, { button, count, signal }). Pass it to your requests, so an action stops when the run stops. keyUp and mouseUp get no signal, because they release what an aborted action held down.

Add the optional methods#

Four methods are optional:

• zoom(region): returns PNG bytes of one region at full size. The region has x, y, width, and height. computer_toolset_20260801 and computer_20251124 turn on zoom when the driver has this method. computer_20250124 does not.
• cursorPosition(): returns the mouse position as { x, y }. Only computer_toolset_20260801 turns on cursor_position, and only when the driver has this method.
• navigate(url): opens the URL. computerUse.tools then has the navigate tool. The tool accepts only http: and https: URLs.
• close(): frees the machine. createComputerUse calls it only on a driver that a factory made.

For example, to add cursorPosition, add this function to app/my-driver.ts. It narrows the JSON from your agent before it returns it:

export async function readCursorPosition(base: string) {
  const response = await fetch(`${base}/cursor`)
  const data: unknown = await response.json()
  const isPoint =
    typeof data === 'object' &&
    data !== null &&
    'x' in data &&
    'y' in data &&
    typeof data.x === 'number' &&
    typeof data.y === 'number'
  if (!isPoint) throw new Error('The agent sent no cursor position.')
  return { x: data.x, y: data.y }
}

Then add cursorPosition: () => readCursorPosition(base) to driver in createMyDriver.

One machine for each thread#

One shared machine serves one conversation at a time. To give each thread its own machine, write a ComputerDriverFactory. createComputerUse calls create on the first action of each thread.

This example asks a pool service for one VM for each thread, and stops the VM when the thread closes:

// app/my-machines.ts
import type {
  ComputerDriver,
  ComputerDriverFactory,
} from '@tanstack/ai-computer-use'

// From app/my-driver.ts above.
declare function createMyDriver(base: string): ComputerDriver

const pool = 'http://localhost:9000'

export const myMachines: ComputerDriverFactory = {
  display: { width: 1920, height: 1080 },
  environment: 'windows',
  capabilities: { zoom: false, cursorPosition: false, navigate: false },
  async create({ threadId }) {
    const machine = `${pool}/machines/${encodeURIComponent(threadId)}`
    const response = await fetch(machine, { method: 'PUT' })
    if (!response.ok) throw new Error(`Start failed: ${response.status}`)
    return {
      ...createMyDriver(`${machine}/agent`),
      close: async () => {
        await fetch(machine, { method: 'DELETE' })
      },
    }
  },
}

Then give the factory to createComputerUse:

import { createComputerUse } from '@tanstack/ai-computer-use'
import type { ComputerDriverFactory } from '@tanstack/ai-computer-use'

// From app/my-machines.ts above.
declare const myMachines: ComputerDriverFactory

const computerUse = createComputerUse({ driver: myMachines })

The factory has these members:

When a thread closes or goes idle, createComputerUse calls close() on the driver of that thread.