dsh-mac-vision

简体中文

A native macOS vision plugin for DeepSeek Harness. It gives text-only models OCR and visual detection for local images, clipboard images, screens, and application windows through Apple Vision.

• Completely free and open source: the plugin uses the MIT License, with no subscription or per-call fees.
• Fully local processing: capture, OCR, and visual detection run on the Mac. Images are not sent to third-party OCR or vision services.
• No third-party model downloads: it uses Apple Vision built into macOS, with no extra vision models or weights to install and maintain.
• No third-party API required: no OCR, Vision, or multimodal-model API key is needed, and there are no such API usage charges.

These statements apply to the visual capabilities provided by dsh-mac-vision. Any cost for the language model used by DeepSeek Harness depends on the user's own Harness configuration.

Quick start

1. Install the plugin

dsh plugin --profile default add dsh-mac-vision

This single command installs both the vision tools and their model-facing usage policy. Do not download, copy, or install SKILL.md separately.

2. Verify and start Harness

dsh --profile default --dump-config
dsh --profile default

On the first vision task, the plugin uses the system Swift compiler to build and cache its local helper. You can then ask the model directly:

Read the title and main content of /absolute/path/to/slide.png.

Or:

Inspect the frontmost window and separate what you directly observe from your interpretation.

Requirements

• macOS
• Node.js 22 or later
• DeepSeek Harness installed through its quick-start guide, with a working dsh command
• Xcode Command Line Tools, or Xcode with a Swift compiler

If no Swift compiler is installed, run:

xcode-select --install

Reading local image files does not require Screen Recording permission. For screen or window capture, macOS may ask you to grant Screen Recording access to the terminal or host application running DeepSeek Harness under System Settings → Privacy & Security → Screen Recording. Restart that application after granting access if necessary.

It is one plugin

Install only dsh-mac-vision; users do not choose between a Plugin and a Skill.

Internally, the plugin contains:

• Vision tools: perform capture, OCR, and Apple Vision detection.
• Bundled Skill: only teaches the model when to call those tools and how to report observations, inferences, and uncertainty.

The Skill cannot perform OCR on its own. It is model-facing guidance that loads automatically with the native plugin, not a second product or installation step.

Capabilities

• Use Apple Vision at no cost, with no third-party vision-model download, API key, or per-call fee.
• Inspect local images, clipboard images, the full screen, the frontmost window, or a selected window.
• Run OCR locally through macOS Vision.
• Return structured evidence with text, confidence, coordinates, alternatives, and review state.
• Automatically crop and upscale small text or notation for a second OCR pass.
• Optionally run image classification, barcode, saliency, people, and cat/dog detection.
• Distinguish direct observations, semantic inferences, conflicting OCR, and detectors that were not run.
• Respect Harness cancellation signals, timeouts, and output-size limits.

The plugin registers two model-facing tools:

• mac_vision_inspect: inspect an image, clipboard, screen, or window.
• mac_vision_list_windows: list visible windows before selecting one.

You normally do not call these tools manually. Describe the task and let the model choose when they are needed.

More examples

Read the text in the clipboard image and clearly mark anything uncertain.

List the visible windows, then inspect the error shown in the browser window.

Check this screenshot for a QR code and extract its visible text: /absolute/path/to/image.png

The model should distinguish:

• Observed: evidence returned directly by OCR or completed detectors.
• Inferred: interpretations based on layout, text placement, or multiple observations.
• Uncertain: conflicting OCR, unverified notation, or detectors that were not run.

Update and uninstall

Update to the latest version:

dsh plugin --profile default update dsh-mac-vision

Uninstall:

dsh plugin --profile default remove dsh-mac-vision

Replace default if you use a different profile name.

Configuration

Most users do not need to change the defaults in cordis.patch.yml:

config:
  timeoutMs: 45000
  maxOutputBytes: 8388608
  defaultMode: fast
  defaultLanguages: []
  defaultRefineText: true
  defaultRefineLimit: 12
  defaultRefineScale: 3
  allowedSources: [file, clipboard, screen, front-window, window]

Override settings in the profile's cordis.patch.yml. Harness patch layers replace the complete config value instead of deep-merging individual keys, so repeat the full config block when overriding it.

For example, allow local files only:

allowedSources: [file]

Troubleshooting

• Swift compiler not found: run xcode-select --install, then retry.
• Files work, but screen or window capture does not: check macOS Screen Recording permission and restart the terminal or Harness host.
• Confirm the plugin is loaded: run dsh --profile default --dump-config and look for dsh-mac-vision, mac-vision-tools, and mac-vision-skill.
• Small text or formulas are inaccurate: ask the model to re-inspect the specific region and preserve uncertainty explicitly.

Local development

This section is for contributors. Normal installation does not require cloning the repository, running pnpm, or building a tarball.

git clone https://github.com/Kevoyuan/dsh-mac-vision.git
cd dsh-mac-vision
pnpm install
pnpm check

When developing from a DeepSeek Harness source checkout, create an overlay with absolute paths:

- insert:
    - id: mac-vision-tools
      name: '/absolute/path/to/dsh-mac-vision/src/index.ts'
    - id: mac-vision-skill
      name: '/absolute/path/to/dsh-mac-vision/src/skill.ts'

Start Harness from its repository:

pnpm dsh web --patch /absolute/path/to/overlay.yml

These Cordis entries belong to the same native plugin. A normal npm installation loads both automatically; users do not configure them separately.

Links

• npm package
• GitHub repository
• DeepSeek Harness plugin installation guide

Feedback and contributing

Open a GitHub issue for bugs or feature requests. Run pnpm check before submitting code changes.

License

MIT