dsh-screen-reader-cbg33695

Gives the agent eyes: it looks at the screen itself, instead of waiting for you to take a screenshot. Plus an exact pixel diff — "did it change, and where" is computed locally, and the model is only asked "what did it become".

Windows only. Requires PowerShell 5.1 with System.Drawing and PrintWindow.

⚠️ Read this first: what the advantage actually is

It does not make vision better. Images reach the model through the model's own pipeline and are squeezed onto the same normalised token grid. The plugin sees exactly as clearly as built-in vision does — no more.

What it adds are two things a model cannot get on its own, and those two happen to decide whether an agent can work by itself:

In one sentence: other plugins look at images for the model; this one gives the agent hands and eyes.

That is its only reason to exist. If you are happy to screenshot manually and paste, it is worth little to you — and a manual screenshot is often better, because you choose the crop.

When you should not use it

Contents

• What it solves
• The two tools
• How images reach the model (this decides how you use it)
• Measured effects
• Boundaries: what it cannot do
• Install
• Upgrading 0.2 to 0.3: breaking changes
• Privacy
• Design notes
• Verification status
• Known issues and plans
• Reporting
• License
• Relation to other plugins

What it solves

Models have no eyes on your screen. So this happens:

• You say "something looks wrong in my interface" and the model can only work from your description
• The model says "click that button" — it has never seen that button
• You change a setting and the model does not know what it became
• You have an agent render an image or restyle a UI and it cannot verify what it just produced

The first three are "cannot see". The fourth is "cannot see and cannot self-verify" — and that is the real bottleneck for automation.

The plugin gives the model two things:

1. Sight: capture the screen (whole desktop / one window / one region) and hand the image itself to the current model, with no second model transcribing it
2. Verification: locate changes between two images exactly, and let the model explain only "what it became"

The two tools

Screen

Comparing two images

There is no "look at one image file" tool here. There used to be see_image, but it was a worse duplicate of the built-in read_image (it added a model-to-model transcription hop). With built-in vision available, that job belongs to the built-in tool, so it was removed.

How images reach the model (this decides how you use it)

The screen tool does not hand the image to another model to transcribe; it returns the image itself as a content block for the current session model to look at. This path is verified.

On the provider side every image is force-normalised, and that decides all usage:

14px patch grid · 3:1 downsample per axis · 384 tokens per image, capped

(The 675×387 row is derived from source constants, not measured. The 346×346 row is measured.)

Normalisation actually happens in two stages, and the first one is measurable: the harness itself caps the request at roughly 640,000 pixels — a 1920×1080 screenshot went out as 1066×600 (639,600 px) — and the provider then squeezes that onto its token grid. Cropping works because a small crop is passed through untouched by that first stage, skipping the 0.555 linear downscale.

Three consequences are therefore necessary, not coincidental:

1. Source resolution does not affect accuracy at full-window scale — two different resolutions land on the same grid
2. Cropping is the only way to buy detail — measured at roughly 2× the effective detail density
3. Raising capture resolution buys no accuracy — a larger source is simply compressed harder

This part is measured, not derived. Same sidebar region, same prompt:

Ground truth came from the session index under ${DSH_HOME}/storages. At full screen, 4 of 5 titles were character-exact and 1 was wrong; after cropping, none were wrong. The full-screen failure mode was exactly visually similar characters (戮/戳 share the same right-hand radical) — precisely what a 0.49 sampling rate predicts.

So the intended usage is: pin the application with window, then zoom with region.

Measured effects

Everything below was measured on real runs, not designed.

Vision

Local pixel diff (the most reliable part)

Cost

Images are cheap: the provider normalises every request image to ≤384 vision tokens regardless of source size. The earlier claim of "about 1000-1500 tokens per look" was wrong.

What is expensive is having a model turn an image into text. On the transcription compatibility path, that model first spends a large reasoning budget (measured 788–6621 chars) before emitting any prose. That is why the default path calls no second model — the image goes straight to the current model.

Boundaries: what it cannot do

This is the most important section in this document.

1. It is a describer, not a gauge

Measured: a 30 px tall colour band was read as "40–45 px" — about 40% error.

• ✅ Reliable: which is taller, which is greyed out, roughly upper-left or lower-right
• ❌ Unreliable: is the gap 8 px, is this spacing 16 or 12

2. Low-contrast differences are the danger zone

Reverse validation works (grey vs saturated blue "disabled button" is easy), but the genuinely hard cases — two similar greys, a 1 px offset, a faint colour shift — are exactly where it misses, or worse: confidently invents one. And "tell me what's wrong with my UI" needs precisely those.

3. Shape semantics are a weak spot

Measured case: a cube edited into a "pointed little house" in a Blender viewport.

• The model saw "a non-standard shape" and said "this object has been edited"
• But it kept calling it a cube and never attributed the mesh edit to the object

Honest caveat: at that camera angle the point was genuinely subtle, so missing it is partly forgivable. But the pattern — not attributing a mesh edit to the object — appears beyond this one case.

4. Resolution is NOT the accuracy lever, but cropping IS

Same Blender window, same prompt, same model — only the resolution changed:

The downscaled one scored better. Conclusion: the provider normalises its input, so at full-window scale the source resolution does not affect accuracy. The mechanism is in the previous section, and the same mechanism explains why cropping works — measured at roughly 2× the effective detail density.

5. One frame only — no time, no causality

• ✅ It can answer: "what is on the screen right now"
• ❌ It cannot answer: "did this dialog appear because I clicked X"

Rolling screen memory (screen_memory / screen_watch) existed to cover this and was removed in 0.3: no real use case ever demonstrated its value, while every recorded frame wrote a file into the attachment store. So today it genuinely sees only the present.

6. The attachment store grows, and there is no cleanup mechanism at all

Every vision call writes a content-addressed file into ${DSH_HOME}/attachments, and the plugin cannot prevent it.

Measured (2026-09-14, this machine's web profile): 215 files / 40.24 MB, accumulating 9 → 25 → 83 → 87 → 13 files per day since 09-10. Its composition:

Why the plugin cannot clean it up — this is not "was not done", it is not possible:

The attachments service describes itself as an "Immutable binary attachment service", and its methods are only validateImage / saveImage(s) / readImage / imageHostPath / readImageRequest. There is no delete, no prune, no gc. And readImage verifies that the bytes still match the recorded reference, so deleting files behind the service's back makes images in past conversations unreadable — those references are persisted in the session log.

This is not specific to this plugin: any image entering a conversation writes here — ones you paste yourself, the built-in read_image, generate_image. The 80 WEBP files in the table above are not from this plugin (it stores PNG only). The real fix belongs at the DSH level (a retention policy for that immutable store), not inside one plugin.

0.2.1 had a vision_storage tool that reported and optionally pruned that directory; 0.3 removed it. Removing it took away no capability: its "prune" was the dangerous half, and a safe prune never existed. To see how big it is, just look at the directory.

The only honest way to reclaim the space: delete ${DSH_HOME}/attachments outright — at the cost of images in every past conversation no longer rendering. That is a trade-off, not a cleanup.

7. Windows only

scripts/capture.ps1 and scripts/imageops.ps1 depend on PowerShell 5.1, System.Drawing, PrintWindow and DwmGetWindowAttribute. On other platforms they fail honestly rather than pretending to succeed.

Install

Option 1 — as a profile bundle (recommended, installed on a real instance and composition-verified)

dsh plugin --profile <profile-name> add <package-or-local-path>
# for a local directory:
#   dsh plugin --profile web add C:\path\to\dsh-screen-reader

dsh plugin forwards its arguments to pnpm and then automatically appends every dependency that declares dsh.bundle.patch to the dsh.profile.bundles layer stack — this package declares it, so no manual configuration is needed.

DSH must be restarted before the row joins the running composition: the bundle layer stack is composed at process start, so a running process never notices a newly installed package.

Check that it landed, without restarting:

dsh --profile <profile-name> --dump-config
# the output should contain:
#   # == dsh-screen-reader
#   - id: screen-reader
#     name: dsh-screen-reader

Then restart DSH and ask any session "which tools do you have related to screens and images?" — it should list two: see_screen and see_diff.

The row is a link: to the package directory (that is what pnpm does for local directory dependencies), so do not delete or move that directory — DSH would fail to find it at startup. To uninstall: dsh plugin --profile <profile-name> remove dsh-screen-reader.

⚠️ The 0.2.0 bundle was broken; 0.2.1 fixes it. Reviewing the code, I found that lib/screen.js and lib/toolbox.js located their PowerShell helpers with new URL('capture.ps1', import.meta.url) — but in the package the JS lives in lib/ while the helpers live in scripts/, so they looked for a lib/capture.ps1 that does not exist. Any bundle install failed on the first capture with The argument '.../lib/capture.ps1' to the -File parameter does not exist. The runtime now handles both layouts (the same source is correct as a preset and as a package), and that check is now an assertion in the generator (tools/build-lib.mjs). If you installed 0.2.0, please upgrade. To be explicit: what is verified is the install and the composed tree. The tool list inside a restarted session is still unconfirmed — that needs a process restart, which I did not perform mid-session.

Option 2 — as an agent preset (verified working)

Put lib/ and scripts/ into a preset directory, wire up agent.cordis.yml, and open a session on that preset. See the DSH docs for preset syntax and relative-path resolution.

Self-check after installing

Which tools do you have related to screens and images?

You should see two: see_screen, see_diff.

• They appear → installed. Then actually call see_screen once to confirm it really returns an image
• They do not → the plugin was not loaded. Try the other install method, or file an issue with your DSH version, profile name and the full error

Upgrading 0.2 to 0.3: breaking changes

0.3 is a large reduction. The tool count goes from 8 to 2.

What to do when upgrading:

1. If a prompt or script of yours names a removed tool, change it. Calling a removed tool returns "unknown tool"; it does not silently degrade
2. see_screen's plugin declaration changed: inject went from ['tools','timer'] to ['tools']. If you install as a preset and wrote your own agent.cordis.yml, you do not need to change anything — inject lives in the plugin source, not in configuration
3. Preset rows need no change: the two plugin rows are still ./plugin/screen.js and ./plugin/toolbox.js; they simply register fewer tools
4. Old versions do not auto-upgrade; the plugin is a local/package install with no update channel

Unchanged: the parameters and return values of the two kept tools, and how cordis.patch.yml inserts the row.

Privacy

This section matters, because screen contents are the most sensitive thing here.

1. Nothing happens unless something captures. 0.3 removed continuous recording, so there is no background capture at all. A screenshot is taken only when you (or the agent) explicitly call see_screen
2. The screenshot goes only to the current session's model. It travels the same channel as images you paste yourself, under the same data policy. The plugin sends nothing to any third party
3. On disk the plugin keeps one image: a fixed path overwritten in place (${DSH_HOME}/vision/screen.png), deleted when the plugin stops
4. But the attachment store is a separate matter: every vision call adds a content-addressed file under ${DSH_HOME}/attachments and nothing can clean it up (measured 215 files / 40.24 MB; see boundary 6)
5. see_diff never captures the screen: it only reads the two image files you point it at
6. Suggestion: call it when needed; there is no longer any reason to leave it "on", because it no longer captures anything by itself

Design notes

Why the plugin modules have zero dependencies

The loader resolves relative specifiers against the preset directory and bare package names against the harness install — a local preset cannot reach @deepseek-ai/dsh-tools. So the plugin deliberately depends on nothing: instead of defineTool(...) it constructs the runtime shape that helper produces, and writes raw JSON Schema (which is why every schema spells out required). It imports only node:fs and node:url.

Cost: no automatic parameter validation from defineTool, so every execute does its own type coercion.

Why half the tools were deleted

The only criterion was: can the model already do this itself?

• It can (look at an image, read text in an image) → delete, use the built-in capability
• It cannot (see the screen, decide exactly which pixels changed) → keep

By that criterion see_image was duplication, the three diagnostics were not capabilities, and continuous recording had no demonstrated value against a certain cost (unbounded attachment growth). The plugin went from 8 tools and ~78 KB of source to 2 tools and ~54 KB.

Why the diff is computed locally

Vision models are poor at fine-grained change spotting and will invent "plausible" differences. A pixel diff is exact, free, and yields bounding boxes. So the split is: compute the diff locally, let the model only explain it.

Why see_screen returns the image instead of text

It used to hand the capture to a second model to transcribe. That path had three problems: an extra layer of transcription error, subjection to that model's reasoning budget (which routinely consumed the entire budget and left the prose empty), and a second billed call. Now the image is returned as a content block for the current model to look at. transcribe: true remains for text-only session models.

Why the prompt puts "subject" first

Measured: if the prompt asks for verbatim text first, the model transcribes text and ignores the picture. After making "answer the subject first, text last" an ordering requirement, judging the selected object went from 0/3 to 2/2.

Verification status

There is still no "92% accurate" benchmark figure here. Every item under "Measured effects" is a concrete case, not a benchmark.

The closest thing to a number: in one comparison over 5 session titles / 50 Chinese characters, full-screen 1920×1080 got 1 character wrong, and a 346×346 crop got 0 wrong. That sample is far too small to call an accuracy rate — all it shows is that the full-screen failure mode is visually similar characters, and that cropping fixes it.

Known issues and plans

What 0.3.2 changed (text-only session models go from broken to working):

• see_screen now decides for itself whether the session model accepts images, and transcribes automatically when it does not. The root cause: before sending, the provider scans messages for an image, and if the model does not declare image in inputModalities it throws LlmError outright (dsh-llm-deepseek L1602-1605) — the whole request fails and the model never sees even the text. And image input is the minority of what a machine may have configured:

  Route	Image input
  deepseek-official/deepseek-v4-flash	❌ text only
  deepseek-official/deepseek-v4-pro	❌ text only
  deepseek-official/deepseek-v4-flash-vision-exp	✅
  local-ollama/qwen3:1.7b / qwen3:0.6b / qwen3-4b-fast	❌ text only

  (Measured on one machine: 1 of 6 routes accepts images.) So in a text-only session this tool is not merely unhelpful — calling it once wrecks that turn, and the agent learns to leave it alone. It now reads {provider, model} from agent.options (via agents.currentInitiator(), verified working) and asks llm.resolveModelInfo(...).inputModalities; when images are not accepted it takes the transcription path.

• Two certain bugs fixed along the way:

  1. The transcription path still sent the image — the attachment was saved before the wantText branch, and render adds an image block whenever imageRef is present. So transcribe: true never actually worked on a text-only model. The transcription path now writes no attachment and returns no image block.
  2. The transcription path always judged itself a failure — ok was written as hasImage && ..., and that path has no imageRef by design, so ok=false and render took its first branch: "看图失败：no capture". A working path was disabled by its own success test. ok is now decided by which path actually ran.

• When the session model cannot be determined, it is treated as "does not accept images." The risk is asymmetric: sending an image to a model that rejects it fails the entire conversation turn, while transcribing at worst costs one look without the original image.

• The result gains visionPath (image / transcribe), sessionModel and note, and the rendered line says plainly that the session model does not accept images and a transcription was used — a degradation has to be stated, not hidden.

What 0.3.1 changed (the capture output contract) — this is a breaking field rename:

• TARGET split into SOURCE / SOURCESIZE / REGION. capture.ps1 used to emit a TARGET that carried both a description of what was captured and a size-looking string like 1920x1080, while SIZE held the artifact's dimensions. So TARGET=virtual screen 1920x1080 could accompany a written file of 1190x486 — anything reading that field as an image size read the wrong number. This is not hypothetical: another session tripped over exactly that during a real acceptance test. Now each of the four lines says one thing: SOURCE (words only, no dimensions), SOURCESIZE (the captured surface), REGION (present only when a crop was used), SIZE (the artifact — the only field describing the output).
• see_screen's output fields follow: target became source + sourceSize + region, and the rendered line now separates the two sizes: 截图范围：virtual screen（区域 …）；源 1920x1080 → 成品 1190x486. If a prompt or script of yours reads target, change it.
• Also fixed a bug I introduced while making that change: $rx is a [double], and an expression starting with a double makes PowerShell treat + ',' as numeric addition, throwing Cannot convert value "," to type System.Double. Now uses -join. The old code only escaped this because it started from a string.
• Verified across all four modes (screen / screen+region / window / window+region), and SIZE was checked byte-for-byte against the real dimensions in the PNG's IHDR.

What 0.3.0 changed:

• see_diff deleted your input images (data loss; fixed): imageops.ps1's retention rule deleted the N oldest files in the output directory no matter who wrote them. So when the two images you wanted to compare happened to live in the same folder as the crop files, retention ran after writing, protected this run's crops, and then ate your a.png / b.png. Silent data loss of exactly the thing that cannot be regenerated. Retention now deletes only files this tool itself produced (diff*_A.png / diff*_B.png), and every call site additionally protects the explicit inputs it was handed. Verified: inputs survive, and the crops still stop growing
• Tool count 8 → 2 (criterion in Design notes). Removed see_image, screen_watch, screen_memory, vision_routes, vision_selftest, vision_storage
• Removed with them: the change fingerprint, rate-limit gate, rolling buffer, the timer dependency, concurrency mutex — and a branch in see_screen that could never be reached (it always called observe with force=true, so those checks were dead code)
• see_screen source 50.3 KB → 35.1 KB; toolbox 27.5 KB → 19.1 KB
• Breaking-change list in the previous section

What 0.2.1 fixed:

• The published ps1 paths were broken (0.2.0's bundle could never work): now resolved at runtime for both layouts (the same source is correct as a preset and as a package), with the generator tools/build-lib.mjs committed to the repo and asserting it
• capture.ps1 -Out with a bare name polluted the current directory: bare names now resolve into ${DSH_HOME}/vision/ and get a .png extension; absolute paths unchanged
• Documentation correction: crop detail density revised down from a derived "2.7×" to a measured ~2×

Plans:

1. Produce a real accuracy benchmark — the biggest remaining gap. It was going to be vision_selftest, but that covered only the diff chain and never see_screen, which is where a number is most needed. So it was removed and the benchmark needs redesigning
2. Rewrite the tool definitions with @deepseek-ai/dsh-tools' defineTool to get parameter validation back
3. Wire up the Config schema (tunables are file-header constants today)
4. Trim the now-unreferenced modes in scripts/imageops.ps1 (calib-draw / calib-bad / storage / prune), which existed for removed tools
5. Shape semantics: try a two-pass "locate the subject, then zoom into it", already proven to help selection accuracy

Known design trade-offs:

• About 40 lines of duplicated vision-routing and error handling between lib/screen.js and lib/toolbox.js. The reason is in the file headers: the loader caches modules by URL, and a relative import would tie the two plugins' lifetimes together
• Diff bounding boxes are 30–45 px larger than the true change (tunable via -Padding / -Dilate / -GridW)
• The time dimension (rolling screen memory) has been abandoned. If a real use case appears, it can be recovered from git history (git show 98306ac5:lib/screen.js)

Reporting

This plugin is experimental and admits to plenty it cannot do. The feedback I want most is "here is where it got it wrong for you", not praise.

Please include:

1. Install method (bundle / preset) and dsh --version
2. The tool you called and its full arguments
3. The complete return value or error (not just "it doesn't work")
4. If an image is involved: attach that image. Without it I cannot tell whether the fault is the plugin or the model

License

MIT. See LICENSE.

Relation to other plugins