dsh-plugin-vision

English | 中文

Give DeepSeek Harness (dsh) the ability to see and create images. Two tools:

• image_analyze: reads an image file through dsh's sandboxed fs service (workspace and permission policy apply), sends it to an OpenAI-compatible vision endpoint (chat/completions with image_url content), and returns a text description or answer.
• image_generate: sends a text prompt to an OpenAI-compatible image endpoint (images/generations), saves the generated image into the workspace, and returns the file path. Before generating, an automatic prompt-engineering pass runs (see below).

The endpoint, models, and credential are all configuration — official OpenAI, relay/proxy services, or self-hosted OpenAI-compatible gateways all work.

Why this plugin exists: dsh's built-in DeepSeek adapter rejects image content (UNSUPPORTED_CONTENT). This plugin delegates "image → text" and "text → image" to dedicated models, so your main agent model needs no image support at all.

Features

• Pluggable like every dsh plugin: appears in the Settings → Plugins → Plugin list, config changes hot-reload
• image_analyze reads through the sandboxed filesystem — workspace and permission policy apply
• OpenAI-compatible: configure baseUrl, models, and API key freely
• image_analyze supports png / jpeg / gif / webp, with configurable size limits and timeouts
• image_generate saves into the workspace (default generated-<timestamp>.png, or a path you choose) and refuses writes outside the workspace root
• Automatic prompt engineering before generation: no extra model is configured or billed — the session's own model (the one dsh is already using, e.g. DeepSeek) understands the request, rewrites it into a structured image prompt (Goal / Main subject / Composition / Visual style / Lighting and color / Details / Text / Constraints), and picks an official output size. If the pass fails for any reason, it degrades gracefully to the original prompt and default size — generation is never blocked.

Requirements

• A running DeepSeek Harness Web profile (install Node.js first, then npx -y @deepseek-ai/dsh web)
• Recognition: a vision-capable chat model behind an OpenAI-compatible API — e.g. gpt-4o, gpt-4o-mini, gpt-4.1
• Generation: an image-generation model — e.g. gpt-image-1, gpt-image-2, dall-e-3
• ⚠️ Generation models cannot read images, and vision chat models usually cannot generate them — configure model and generateModel separately

🚀 Don't want to do it yourself? Let your dsh agent install it

Copy the whole block below to your agent (the one you are chatting with right now) — it will do the installation for you:

Please install the dsh-plugin-vision image plugin for me:

1. Run git clone https://github.com/WUDIFIVE/dsh-plugin-vision.git (clone into a temp directory)

2. Copy the whole dsh-plugin-vision folder into ~/.dsh/profiles/web/dsh-plugin-vision (if the target exists, remove the old copy first)

3. Append the following to ~/.dsh/profiles/web/cordis.patch.yml (put the key in apiKey, not apiKeyEnv):

   - insert:
       - id: vision
         name: './dsh-plugin-vision/lib/index.js'
         config:
           apiKey: sk-your-key
           baseUrl: https://api.openai.com/v1
           model: gpt-4o
           generateModel: gpt-image-2
           imageSize: 1024x1024
           detail: auto
           timeoutMs: 120000
           maxImageBytes: 15728640
           maxTokens: 1024
   

4. Check Settings → Plugins → Plugin list shows vision enabled

5. If writing to ~/.dsh needs permission approval, prompt me to allow it

💡 The agent may ask for permission to write outside the workspace — just approve it. Config changes usually hot-apply; if not, restart dsh web.

Or, if pnpm is already set up, give your agent this one-liner:

dsh plugin --profile web add github:WUDIFIVE/dsh-plugin-vision

Then have the agent append an entry with the short name name: dsh-plugin-vision to ~/.dsh/profiles/web/cordis.patch.yml (see "Standard install" below).

Installation (manual)

macOS / Linux

1. Install the plugin package into the profile (standard path, requires pnpm):

dsh plugin --profile web add github:WUDIFIVE/dsh-plugin-vision

Once published to npm: dsh plugin --profile web add dsh-plugin-vision. Enable pnpm with corepack enable, or npm install -g pnpm.

2. Add the entry to your profile patch — append to ~/.dsh/profiles/web/cordis.patch.yml:

⚠️ Common mistake (many people get this wrong): apiKey holds the key itself; apiKeyEnv holds an environment-variable NAME (e.g. OPENAI_API_KEY). Use one of the two: either hard-code the key in apiKey: sk-xxx, or leave apiKey empty and put the key into the environment / the credential store (web Models page). Do not put the key string into apiKeyEnv — the plugin will treat it as an environment-variable name, fail to find it, and report VISION_CREDENTIAL_MISSING.

- insert:
    - id: vision
      name: dsh-plugin-vision
      config:
        apiKey: sk-your-key        # ← the key goes here (or leave empty and use env/credentials)
        # apiKeyEnv: OPENAI_API_KEY   # ← this is an env-var NAME, not the key! pick one
        baseUrl: https://api.openai.com/v1
        model: gpt-4o              # recognition model (vision chat model)
        generateModel: gpt-image-2 # generation model (text-to-image model)
        imageSize: 1024x1024       # default generation size
        detail: auto
        timeoutMs: 120000
        maxImageBytes: 15728640
        maxTokens: 1024

3. Restart the Web UI (or let config hot-reload apply):

dsh web

Open Settings → Plugins → Plugin list: you should see vision as an enabled entry.

Windows (PowerShell)

Same three steps, with Windows paths:

# 1. install (requires pnpm: corepack enable, or npm install -g pnpm)
dsh plugin --profile web add github:WUDIFIVE/dsh-plugin-vision

# 2. append the entry above to:
#    $HOME\.dsh\profiles\web\cordis.patch.yml
#    (put the key in apiKey, not apiKeyEnv)

# 3. restart
dsh web

No-pnpm alternative (manual copy)

If you cannot install pnpm, copy the plugin folder into the profile directory and reference it by relative path:

# macOS / Linux
cp -R dsh-plugin-vision ~/.dsh/profiles/web/

# Windows PowerShell
Copy-Item dsh-plugin-vision -Recurse $HOME\.dsh\profiles\web\dsh-plugin-vision

Then use name: './dsh-plugin-vision/lib/index.js' in the patch entry above instead of dsh-plugin-vision. The plugin's dependencies resolve from the profile's hoisted node_modules, so no extra install is needed.

Configuration

API key resolution order: apiKey config → credential service (what the web Models page writes) → apiKeyEnv environment variable.

Usage

Recognition (image_analyze)

The agent automatically calls it when it needs to understand an image. Tool arguments:

• path (required): image file path — absolute, or relative to the workspace root. Supported formats: png, jpeg, gif, webp.
• question (optional): a specific question about the image; when omitted, the image is described in detail.

Example prompts for your agent:

Look at screenshots/login-bug.png — what error is shown, and what is the state of the form fields?

Read scans/receipt.jpg and extract the total amount and date.

看一下工作区里的 docs/架构图.png，用中文总结里面的模块关系

Note: images uploaded directly in chat are not visible to the tool (and dsh's DeepSeek adapter rejects image uploads anyway). Save the image to disk first, then pass its path.

Generation (image_generate)

The agent automatically calls it when you ask for an image. Tool arguments:

• prompt (required): the user's image request in natural language — no need to hand-write an English prompt yourself.
• output_path (optional): where to save the image (relative to the workspace; must stay inside the workspace). Defaults to generated-<timestamp>.png.
• size (optional): output size for this call, overriding automatic selection; valid values are the allowedSizes (official size list).

Built-in prompt engineering: before generation, the plugin asks the session's own model (the one dsh is already using, e.g. DeepSeek — no extra configuration or billing) to rewrite prompt into the eight-section structured image prompt and to pick an aspect ratio from allowedSizes — 1:1 for avatars/icons, 1536×1024-class sizes for horizontal banners, 1024×1536 or 2160×3840 for vertical posters/wallpapers, 2048-class sizes for high-resolution work. The result carries an enhanced flag; on any failure the pass degrades to the original prompt and the default size.

Example prompts for your agent:

Generate an image: a yellow cartoon cat wearing a pink tie, standing in a sunny office, save it to docs/logo.png

Draw a Chinese solar-term poster about "末伏" (last dog days of summer) with sunflowers and a setting sun, vertical layout

The generated file lands in the workspace; ask the agent for the path. You can then run image_analyze on your own generated image — recognition and generation close the loop.

How it works

image_analyze(path, question?)                     image_generate(prompt, output_path?, size?)
   │                                                       │
   ├─ ctx.fs.resolve/stat/readBytes                         ├─ [prompt engineering] ctx.llm.stream()
   ├─ detect media type (png/jpeg/gif/webp magic bytes)     │   session's own provider/model (the one dsh uses)
   ├─ data:image/...;base64                                 │   rewrites the request into a structured prompt
   └─ POST {baseUrl}/chat/completions                       │   and picks an official size (allowedSizes)
        messages: [{role:user, content:[                     │   ↓ on failure: original prompt + default size
          {type:text, text:question},                        ├─ POST {baseUrl}/images/generations
          {type:image_url, image_url:{url:dataUrl, detail}}  │    {model, prompt(enhanced), n:1, size, response_format:"b64_json"}
        ]}]                                                  ├─ decode base64 → detect type → pick extension
        → choices[0].message.content                         └─ write into the workspace (default generated-<timestamp>.png)

Development & testing

git clone https://github.com/WUDIFIVE/dsh-plugin-vision.git
cd dsh-plugin-vision
npm install        # installs the @deepseek-ai/* dependencies for local testing
node test/smoke.mjs

The smoke test runs 25 checks — module shape, both tool registrations, recognition request body (base64 data URL, model, question), the prompt-engineering pass (session-model driven, structured rewrite, automatic size selection, explicit-size override, illegal-size fallback, failure degradation, no-session-model degradation), generation request body and file write, out-of-workspace path rejection, credential resolution order, and error paths — with a stubbed fetch and ctx.llm, so no real API key is needed.

Troubleshooting

Common symptoms and fixes:

• "The interface does not currently support this model" / model not found: wrong model id, or the model does not support the endpoint you are using (e.g. a text-to-image model like gpt-image-2 placed in the recognition model). Recognition needs a vision chat model (e.g. gpt-4o); generation needs a text-to-image model (e.g. gpt-image-2). Don't mix the two fields.
• VISION_CREDENTIAL_MISSING with an sk-... value in the message: the key was written into apiKeyEnv — move it to apiKey (see the warning above).

License

MIT