Why I Give Each AI Agent Its Own Screen, Browser, and Keyboard

What if every AI agent had its own computer?

Not a shared browser. Not a headless tab. A full desktop with its own screen, its own Chrome instance, its own extensions, its own session cookies.

I built this. It's called Deck.

The problem with shared browsers

When one AI agent uses a browser, it works fine. When three agents need browsers simultaneously:

• Sessions collide -- one agent logs out another
• Extensions interfere -- ad blockers break automation scripts
• Cookies leak -- Agent A's LinkedIn session ends up in Agent B's context
• No isolation -- a crashed tab takes down everything

Headless browsers solve some of this. But headless Chrome can't run extensions, can't handle complex login flows, and can't be visually debugged.

The idea

Give each agent a virtual display (Xvfb), a window manager (Fluxbox), a Chrome instance with its own profile, and a VNC server for visual access.

From the outside, it looks like this:

Deck 0:  Display :20  |  Chrome (profile A)  |  VNC :3200
Deck 1:  Display :21  |  Chrome (profile B)  |  VNC :3201
Deck 2:  Display :22  |  Chrome (profile C)  |  VNC :3202

Three isolated desktops running in parallel. Each agent controls its own.

How agents interact with their desktop

Three levels, from fastest to most powerful:

1. API commands -- navigate, click, type, take screenshot. Fast, reliable, works for most tasks.

POST /api/docks/0/browser/navigate  {"url": "https://example.com"}
POST /api/docks/0/browser/click     {"selector": "#login-btn"}
POST /api/docks/0/browser/type      {"selector": "#email", "text": "..."}
GET  /api/docks/0/browser/screenshot

2. CDP (Chrome DevTools Protocol) -- full browser control. Execute JavaScript, intercept network requests, manipulate DOM. For when the API isn't enough.

3. Vision + coordinates -- the agent looks at a screenshot, finds what it needs, and clicks by screen coordinates. For legacy apps, CAPTCHAs, or anything that resists automation.

POST /api/docks/0/find_on_screen   {"query": "Submit button"}
POST /api/docks/0/click_on_text    {"text": "Sign In"}

The agent tries API first. If that fails, CDP. If that fails, vision. Escalation, not fallback.

Live VNC preview

Each deck has a VNC stream. In my HQ dashboard, I can watch what each agent is doing in real-time -- actual browser windows, mouse movements, page loads.

This turned out to be the most useful debugging tool. When an agent is stuck, I don't read logs. I watch its screen.

Chrome extensions

Each deck runs a real Chrome instance with extensions. The extension connects to the Deck server via a bridge, allowing two-way communication.

Why this matters: some sites detect headless browsers and block them. A real Chrome with a real extension profile behaves like a real user. Anti-bot systems see a normal browser session.

What I use it for

• X Analytics -- an agent monitors Twitter feeds, collects engagement data, suggests replies
• Directory registration -- an agent fills registration forms on business directories (one of those things that's too complex for simple HTTP automation)
• Testing -- running parallel browser sessions to test web applications
• Research -- agents browse sites, take screenshots, extract data

The ugly parts

Resource hungry. Each deck runs Xvfb + Chrome + VNC + bridge. That's ~300MB RAM per deck. On a server with 32GB, I can run about 8 decks comfortably.

Chrome is unpredictable. It crashes, shows recovery banners, updates itself, changes behavior between versions. I've spent more time debugging Chrome quirks than building features.

Extensions are fragile. Chrome's Manifest V3 service workers die randomly. The extension needs to reconnect, and sometimes that reconnection fails silently. I wrote a cleanup system that detects and removes stale extension entries on every launch.

Was it worth it?

For anything that requires a real browser session -- authenticated sites, extension-dependent workflows, visual verification -- yes. Headless browsers can't do what Deck does.

For simple web scraping or API calls -- no, use curl or a headless fetch. Deck is overkill.

The right tool for the right job. Sometimes that tool is a full virtual desktop.