dsh-eval

English | 中文

Agent Evaluation Platform for deepseek-harness.

Run benchmarks against headless dsh profiles, harvest persisted session logs as traces, fold automatic metrics, grade task success and tool selection, and report or compare runs — one benchmark.yaml in, one JSON run + Markdown report out.

The dsh ecosystem already has observability and debugging tools (dsh-trace, dsh-tps, dsh-context-doctor). dsh-eval fills the missing slot: an evaluation platform.

Highlights

• dsh eval run benchmark.yaml — orchestrate one headless dsh subprocess per case × trial
• Trace harvesting from persisted session logs (everything a model sees is reconstructable from the log)
• Automatic metrics: task success, tool success, tool-selection accuracy, steps, tokens, latency, cost, retry, invalid tool calls, context usage
• Scripted grading: expected.tool (tool-selection accuracy) and expected.check (task success)
• LLM judge: final-answer score and hallucination flags from a judge model
• Subagent trace merging: child session logs fold into the trial metrics
• Paired A/B: same-case win/lose/tie statistics across two runs
• Paired inference: exact sign test over discordant pairs, seeded bootstrap 95% intervals on every mean delta
• Parallel trials: concurrency / --concurrency runs several case × trial executions at once, results still ordered
• Keyless replay: record once with a key, replay in CI from recorded logs
• Cross-harness import: dsh eval import codex|claude-code <log> --out run.json
• dsh eval report run.json — Markdown report with per-trial scores and pooled rates
• dsh eval compare v1.json v2.json — signed B - A comparison table

Status

dsh-eval@0.4.0 · 129 tests · 100% statement/branch/function/line coverage on src · typecheck clean · CI on Linux and Windows.

Installable since 0.3.1: the 0.3.0 tarball omitted the runtime modules its entry imports, so every install failed with ERR_MODULE_NOT_FOUND. See CHANGELOG.md.

Quick start

Install the package directly:

pnpm add dsh-eval
dsh plugin --profile eval add dsh-eval

The npm package targets the official @deepseek-ai/* releases (0.1.0-rc.6 peers), and this workspace resolves the same versions, so a clone builds against exactly what consumers install — no harness checkout to link:

git clone https://github.com/hccccc01333/dsh-eval.git
cd dsh-eval
pnpm install
pnpm --filter dsh-eval build
pnpm --filter dsh-eval test

Running benchmarks needs a dsh launcher; pnpm test does not (the suite is keyless, with a fake dsh and recorded session-log fixtures):

dsh plugin --profile eval add dsh-eval
dsh eval run benchmark.yaml --out eval-run.json
dsh eval report eval-run.json
dsh eval compare eval-v1.json eval-v2.json
dsh eval import codex ~/.codex/sessions/.../session.jsonl --out codex-run.json

Benchmark document

name: skill-regression
model: deepseek-v4
profile: headless
command: [dsh]
trials: 3
timeoutMs: 600000
seed: 42
cases:
  - id: fix-tests-001
    prompt: Fix the failing tests in this workspace.
    workspace: ./fixtures/fix-tests
    expected:
      tool: bash
      check: ./check.sh
pricing:
  deepseek-v4:
    inputUsdPerMTokens: 0.27
    cacheReadUsdPerMTokens: 0.07
    cacheWriteUsdPerMTokens: 0.27
    outputUsdPerMTokens: 1.10

Each trial runs in a private temp workspace with an isolated DSH_HOME and non-interactive permissions. The primary session log becomes the trial's trace; scripted grading pools taskSuccess and toolSelectionAccuracy into run-level rates. See packages/eval/README.md for the full field reference.

Metrics

CLI

Roadmap

• Per-arm leaderboards over paired trials
• Multiple-comparison correction across the metrics a report prints
• Web UI dashboard for run reports and comparisons

Repository layout

packages/eval/    plugin bundle + CLI app + tests
scripts/          harness-closure.mjs — optional local-harness workspace closure

dsh eval run also takes --trials, --concurrency, --dsh and --profile to override the benchmark document, and dsh eval import takes --case-id. Replay runs need a dsh built from harness source; everything else runs against the released peers.

Releasing

Bump version in packages/eval/package.json and add a CHANGELOG section, then push a matching tag:

git tag v0.4.0 && git push origin v0.4.0

.github/workflows/release.yml re-runs the gates and publishes to npm through trusted publishing (OIDC) with a provenance statement — no token on any machine. That path needs a one-time entry in the npm package settings (Trusted Publisher → GitHub Actions, workflow release.yml); the file's header has the exact fields.

Without that entry, publish by staging and approving in the browser, which is the only route when the account's second factor is a security key (the npm CLI has no way to answer an OTP it cannot generate):

cd packages/eval && npx npm@11 stage publish
# then approve the staged package at npmjs.com -> Staged Packages

License

MIT