dsh-chrome

A DeepSeek Harness plugin that gives the agent model-facing tools to drive a real Chrome browser over the Chrome DevTools Protocol (CDP).

10 tools covering navigation, JS execution, page reading (text and structured), clicking, scrolling, typing, keyboard input, screenshots, and tab/browser lifecycle — a complete browser automation surface for AI agents.

Authoring follows the DSH adding-a-tool and plugin packaging guides. Tools are defined with defineTool; the Chrome lifecycle lives in a ctx.chrome Cordis Service.

Tools

Page-activity auto-hints

The observatory (src/observatory.ts) ring-buffers network + console activity per session (500 entries per stream by default). Reading tools (chrome_navigate, chrome_snapshot, chrome_page_snapshot) append a [page activity: …] line when unread console errors or failed requests exist, so the model discovers page failures without polling. chrome_console_read / chrome_net_list advance the read cursors.

Each tool registers a systemPrompt.section guidance entry and a UI card presenter.

Install

From GitHub

dsh plugin --profile web add github:gemone/dsh-chrome

pnpm ≥10 blocks git-hosted packages' prepare scripts by default. On first install, pnpm prints an error with the package key — add it to the profile's pnpm-workspace.yaml:

onlyBuiltDependencies:
  - dsh-chrome

Then re-run the add. This file lives at $DSH_HOME/profiles/web/pnpm-workspace.yaml.

Pin a commit (github:gemone/dsh-chrome#<sha>) so a later push cannot change what builds on your machine.

From a local checkout

cd dsh-chrome && pnpm install && pnpm build && cd ..
dsh plugin --profile web add ./dsh-chrome

Local add uses link: and does not run prepare, so you must pnpm build first.

From npm or a tarball

dsh plugin --profile web add dsh-chrome
dsh plugin --profile web add ./dsh-chrome-0.4.0.tgz

No build permission needed — lib/ is pre-built.

Verify and boot

dsh --profile web --dump-config   # a "# == dsh-chrome" layer with the chrome-tool row
dsh --profile web

Remove with dsh plugin --profile web remove dsh-chrome.

Configuration

All options live under the chrome-tool: section. Every field is optional with sensible defaults.

Example profile patch (cordis.patch.yml):

- id: chrome-tool
  config:
    headless: false           # visible Chrome window
    port: 9333
    snapshotMaxChars: 100000

Config edits to cordis.patch.yml hot-apply — no profile restart needed (DSH watches the file transactionally).

Connect to an already-running Chrome

google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug

- id: chrome-tool
  config:
    launch: false
    cdpUrl: "http://127.0.0.1:9222"

Reading the interface without images

Not every route can see images, and screenshots are not the only way to read a page. The plugin gives a text-only model three interface-acquisition paths, and can optionally delegate screenshots to a vision model:

1. chrome_page_snapshot (preferred, use-browser-style). One call returns the page as a structured element tree — roles, accessible names, input values/states — where every interactive element carries a [ref=eN] id:

   - heading "Sign in" [ref=e1]
   - textbox "Email" value="" [ref=e2]
   - textbox "Password" [ref=e3]
   - button "Sign in" disabled [ref=e4]
   - link "Forgot password?" href="/reset" [ref=e5]
   

   Act on what you saw by ref — chrome_click { "ref": "e4" }, chrome_type { "ref": "e2", "text": "a@b.c" }, chrome_scroll { "ref": "e1" } — no CSS selector authoring. Refs are invalidated by navigation (and reconnection); acting on a stale ref fails with a "take a fresh snapshot" error instead of clicking something stale.

2. chrome_snapshot with format: "markdown". The page converted to GitHub-flavored Markdown — headings, links with absolute URLs, emphasis, nested lists, fenced code blocks, GFM tables, images. Ideal for archiving a page's content or answering "what does this page say" in a form both models and humans read naturally. Cross-origin iframes (unreachable from the top frame) are emitted as <!-- iframe: src --> markers. format: "text" (the default) returns the plain rendered visible text.

3. chrome_evaluate — arbitrary DOM queries for anything the structured tree does not surface. When an expression returns a non-JSON value (DOM node, function), the result hint teaches the JSON.stringify(...) idiom.

chrome_screenshot tries the image path first: on an image-capable route the capture enters context as an image block. On a text-only route with a configured screenshotAnalysis model, the capture is durably saved and described in text by that model (steer it per call with the tool's prompt argument). On a text-only route without one, it refuses before capturing, with an error that points at the three text tools above.

- id: chrome-tool
  config:
    screenshotAnalysis:
      provider: openai          # any registered provider route
      model: gpt-4o-mini        # any model that declares image input
      # prompt: custom instruction for the description (optional)
      # maxTokens: 1024         # output cap for the description (optional)

For the full map of what can fail (launch, endpoint, protocol, page-level, model-route) and how each failure recovers, see docs/failure-modes.md.

Security

Driving a real browser is shell-equivalent power: it can reach the local network, run arbitrary page JavaScript, read rendered content, and exfiltrate data. This plugin is not a sandbox.

• Prefer headless + ephemeral userDataDir. Warn before attaching to a profile with real credentials.
• Enforce allow/ask via tools/pre-execute (deployment concern); the plugin registers tools and does not invent policy.
• Chrome is a child process outside any agent sandbox. Only install plugins you trust.

Architecture

src/
├── index.ts          plugin entry: name/inject/Config(schemastery)/apply
├── service.ts        ChromeRuntime Service → ctx.chrome (lifecycle + CDP orchestration)
├── cdp.ts            CDP client over ws (flattened sessions, abort-aware)
├── launcher.ts       executable discovery, Chrome spawn, /json/version polling
├── page-walker.ts    in-page DOM walker: element tree + [ref=eN] → selector map
├── markdown.ts       in-page DOM→Markdown converter (chrome_snapshot format)
├── analysis.ts       one-shot vision-model call that describes a saved screenshot
├── types.ts          error codes + shared types
└── tools/            10 defineTool consumers
    ├── navigate.ts     chrome_navigate
    ├── evaluate.ts     chrome_evaluate
    ├── snapshot.ts     chrome_snapshot (text)
    ├── page-snapshot.ts chrome_page_snapshot (structured tree + refs)
    ├── click.ts        chrome_click (selector or ref)
    ├── screenshot.ts   chrome_screenshot (image path + analysis path, gated)
    ├── scroll.ts       chrome_scroll (selector or ref)
    ├── type.ts         chrome_type (Input.insertText, focus by selector/ref)
    ├── press-key.ts    chrome_press_key (Input.dispatchKeyEvent)
    ├── ref-target.ts   shared selector/ref argument resolution
    ├── net.ts          chrome_net_list / chrome_net_body (Network events + getResponseBody)
    ├── console.ts      chrome_console_read (Runtime/Log events)
    └── close.ts        chrome_close (Target.closeTarget / dispose)

• One bundle, provider-shaped seam. ChromeRuntime is ctx.chrome; its interface allows a future split into capability seam + provider + consumer (mirroring ctx.web).
• Flattened CDP sessions. One browser-level WebSocket drives a page target via Target.attachToTarget({ flatten: true }).
• Connected-browser state. After closeTab(), the browser connection stays alive; the next tool call auto-creates a fresh page target — no Chrome restart needed.
• Ephemeral page refs. chrome_page_snapshot keeps a ref → selector map inside the runtime; navigation clears it, and ref-based tools fail loudly on stale ids.
• Lazy, fiber-scoped lifecycle. Chrome starts on first use; torn down (socket closed, child killed) with the owning plugin fiber. HMR-safe.
• Caller-signal honored. Every operation forwards exec.signal; cancellation stops the wait, not Chrome.

Requirements

• DeepSeek Harness ^0.2.0-rc.2 (attachment admission limits and message-source extensibility this plugin builds on)
• Chrome or Chromium on the host (or a reachable CDP endpoint)
• Node ≥20

Development

pnpm install
pnpm run typecheck           # tsc --noEmit
pnpm run build               # tsdown → lib/index.js + lib/index.d.ts
pnpm test                    # unit + plugin tests (no Chrome needed)
DSH_CHROME_INTEGRATION=1 pnpm run test:integration  # real headless Chrome

License

MIT