dsh-computer-use-windows

Experimental Windows computer-use bundle for DeepSeek Harness, built around window-scoped perception, text-grounded actions, and post-action verification.

The repository grew out of a real desktop-automation failure mode: coordinate-only control was brittle when screenshots included unrelated windows, OCR positions drifted, or a click silently landed on the wrong UI state. The implementation therefore treats every action as an observable state transition rather than a blind coordinate command.

中文简介：面向 DeepSeek Harness 的 Windows computer-use 实验插件。核心是目标窗口绑定、OCR 文本定位、点击后验证与失败重试；视觉模型是可选项，纯 OCR 模式不需要外部 VLM。

Status

Experimental alpha. The repository contains a real DSH plugin wrapper (plugins/index.js), helper runtime (helper/cu.ps1), bundle patch, skill documentation, local health checks, and hosted Windows static CI. It is suitable for development and controlled testing, but the project does not yet claim production-grade unattended desktop automation.

The remaining release gate is a clean-install / real-DSH validation matrix on representative interactive Windows configurations.

Design invariants

Implemented surface

The DSH-facing tool registration lives in plugins/index.js; the Windows implementation is kept in helper/cu.ps1 so platform-specific mechanics remain isolated from the host adapter.

Modes

OCR-only

{
  "vision": {
    "enabled": false
  }
}

No screenshot is intentionally sent to a remote vision model in this mode.

Optional vision provider

{
  "vision": {
    "enabled": true,
    "provider": "openai-compatible",
    "base_url": "https://your-vlm.example.com/v1",
    "api_key_env": "MY_VLM_KEY",
    "model": "your-model"
  }
}

The API key is referenced by environment-variable name; it is not stored in the repository configuration.

Local interactive smoke check

Requirements:

• Windows 11 recommended;
• PowerShell 7.4+;
• Node.js 20+ for the DSH plugin surface;
• Windows OCR language packs for OCR-dependent workflows.

Run on the target Windows workstation:

./scripts/check-health.ps1

The script exercises the helper health path and window enumeration. Missing OCR language support is reported as a warning rather than silently treated as available.

For a direct helper call:

$env:CU_ARGS = '{"cmd":"screen"}'
& ./helper/cu.ps1

Hosted CI

GitHub Actions runs on windows-latest and deliberately performs only deterministic checks that are valid on a hosted Windows Server runner:

1. JavaScript syntax for the DSH plugin wrapper;
2. PowerShell parser correctness for the helper and diagnostic entry points;
3. package / bundle / skill entry-point existence.

The full scripts/check-health.ps1 path is not treated as hosted CI because it depends on Windows Runtime/OCR availability and an interactive desktop session. Those properties differ from the target Windows workstation and must be recorded separately as release evidence.

Repository map

.
├── plugins/index.js              # DSH-facing tool adapter
├── helper/cu.ps1                 # Windows implementation
├── skills/computer-use-windows/  # agent-facing usage contract
├── scripts/check-health.ps1      # local interactive smoke/diagnostic entry point
├── docs/
│   ├── design.zh.md              # architecture and config design
│   ├── experiment-findings.zh.md # failure analysis from the original workflow
│   └── research-plan.zh.md       # comparison / validation questions
├── cordis.patch.yml              # DSH bundle composition
└── package.json

Release-readiness work

Before calling this stable, the project should demonstrate:

• clean installation against a pinned current DSH release;
• at least one reproducible OCR-only workflow on Windows 11;
• DPI scaling checks (100% / 125% / 150% where practical);
• English and Simplified Chinese OCR language-pack behavior;
• explicit failure behavior when the target window disappears or verification cannot be satisfied;
• documentation of which actions are deterministic and which depend on a configured VLM.

Safety boundary

This software can inject mouse and keyboard input into desktop applications. Use it only on systems and applications you are authorized to operate. Target-window checks reduce accidental interaction with unrelated windows but do not make arbitrary desktop automation risk-free.

If vision is enabled, screenshots may be transmitted to the configured endpoint. Use OCR-only mode when screenshots must remain local.

License

MIT. See LICENSE.