dsh-fetch-third-party

中文 | English

A safe third-party web fetching plugin for DeepSeek Harness (DSH) — lets AI fetch full web pages on its own when search snippets are not enough to answer.

Introduction

This plugin adds safe web fetching to DeepSeek Harness (dsh): fetching is delegated to a third-party service you choose or host (Tavily / Jina Reader / Firecrawl / custom contract-v1 services). The local machine never connects to the target URL directly, so there is no SSRF attack surface.

Core capabilities:

• Autonomous fetching: the model first searches with the official web_search; when snippets are not enough to answer, it automatically calls web_fetch_url to fetch the full page — no user prompting needed.
• Courier model: the target URL is fetched only by the configured third-party service; the local machine makes zero outbound connections to arbitrary URLs (no SSRF surface).
• Multiple providers + routing: Tavily / Jina Reader / Firecrawl / custom services (contract v1, multiple instances), with automatic fallback from the primary to the fallback provider.
• Single API-key write point: keys live only in the managed credentials store (0600); the card only shows "configured / not configured" and never echoes the key.
• Per-session budget: 10 fetches per session by default; further calls are refused with a clear message.
• GUI settings card: test connection / local proxy / custom provider management / clear key.
• Auto-managed local Crawl4AI stack: when the primary or fallback provider is a loopback custom provider, the plugin starts the container and the wrapper process automatically, with a real-time watchdog for self-healing.
• Fetch cache: repeated fetches of the same URL within the TTL are served from memory — no third-party quota used, no session budget consumed.
• SSRF defense-in-depth: targets on loopback / private / reserved addresses requested by the model are rejected before forwarding (important when a self-hosted Crawl4AI shares the local network).
• Dynamic fallback chain: ordered provider chain with quota/failure cooldown (exponential backoff) — failing providers degrade automatically and recover when the cooldown expires, not just a fixed primary+fallback pair.
• Structured output: web_fetch_url returns title / heading outline / links / word count / reading time alongside the body, making the evidence easier to cite and verify.

How it works

User question
  → model calls web_search (official search)
    → snippets enough? → yes → answer directly
    → no → model autonomously calls web_fetch_url({ url })
      → ctx.web.fetch → third-party provider
        → third-party service (Tavily / Jina / Firecrawl / custom) fetches the target page
        → returns the content → model answers from the full text

Directory layout

fetch/
├── package.json        # dsh.bundle + dsh.client manifest
├── cordis.patch.yml    # patch layer
├── docs/               # contract v1 / self-hosted crawler research
├── scripts/            # verification scripts + contract-v1 wrapper + launch scripts
├── tests/              # vitest unit tests
└── src/                # source (Host + Client halves)

Installing into DeepSeek Harness

Prerequisites: DeepSeek Harness (dsh) and pnpm.

Method 1: git install (recommended)

dsh plugin --profile web add https://github.com/tallahandsome-ux/dsh-fetch-third-party.git

pnpm v11 first-install note: pnpm v11 blocks build scripts of git dependencies by default; the first run may fail with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED. The error prints the exact key to allow; add it to the profile pnpm-workspace.yaml and re-run, e.g.:

# %USERPROFILE%\.dsh\profiles\web\pnpm-workspace.yaml
allowBuilds:
  "dsh-fetch-third-party@git+https://github.com/tallahandsome-ux/dsh-fetch-third-party.git#<commit>": true

After installing, restart dsh web; the "Web fetch (third party)" card appears under Settings → Plugins → Plugin configuration.

Method 2: local build then install

For when you already have a local copy (e.g. a git clone you want to modify).

Key point: file: must point to the full absolute path of the clone directory — the folder that directly contains package.json. Not the repo name, not a file, not the parent directory.

git clone https://github.com/tallahandsome-ux/dsh-fetch-third-party.git
cd dsh-fetch-third-party
pnpm install
pnpm build
dsh plugin --profile web add file:D:\your-folder\dsh-fetch-third-party

Replace D:\your-folder\dsh-fetch-third-party with the actual directory you cloned to. For example, if you cloned into D:\plugins:

dsh plugin --profile web add file:D:\plugins\dsh-fetch-third-party

How to tell it is correct: the path should directly contain package.json, src/, tsdown.config.ts, scripts/, etc.; if Test-Path passes and package.json is visible, it is correct. A wrong path (missing directory, or pointing at a file) makes dsh plugin add fail or install nothing.

If you previously installed an old version, restart dsh web after installing; if the card does not refresh, run pnpm install in the profile directory (%USERPROFILE%\.dsh\profiles\web) and restart again.

Verifying the install

1. Restart dsh web and hard-refresh the browser (Ctrl+F5).
2. Settings → Plugins → Plugin configuration → "Web fetch (third party)" card.
3. Pick a provider (e.g. Jina Reader, keyless) and click "Test connection" — it should succeed.
4. Start a new session; the model can fetch full pages via the web_fetch_url tool.

API Key Security (required reading)

This plugin handles API keys under two hard constraints:

1. Keys are never read back in plaintext: a key lives only in the DSH-managed vault ~/.dsh/.credentials.yaml (written with mode 0600, readable by the current user only). The only read in code is to build the Authorization: Bearer <key> header when a request goes out — no bridge endpoint, GUI card, log line, or error message ever returns the key value; the settings card only shows a configured / not configured boolean (apiKeyConfigured) and never echoes the key itself.

2. If a key must ever be displayed, it must be masked: any future feature (debug output, diagnostics panel, test echo) that shows a key must not print it in full — mask it to a few leading/trailing characters plus *** (e.g. tvly-dev-4y0S…FUb3C35). A full key appearing in any log, error, or response body is treated as a defect (bug).

Additional recommendations:

• Do not put keys in .env files or config files that get committed, and do not paste keys into chat transcripts — once a key appears in a conversation, treat it as compromised.
• If a key is suspected of leaking (logs, chats, screenshots), rotate it immediately at the provider console (issue a new key, revoke the old one).
• (Optional, stricter) tighten the vault file ACL: icacls "C:\Users\<you>\.dsh\.credentials.yaml" /inheritance:r /grant:r "%USERNAME%:(F)" — a dsh process running as the same user is unaffected.

Usage

Switching providers

Settings → Plugins → Plugin configuration → Web fetch (third party) card:

1. Provider dropdown: Tavily / Jina Reader / Firecrawl / custom providers
2. API Key: written to the managed credentials store (0600), only shows "configured / not configured"; Jina may be left blank (anonymous works), Tavily / Firecrawl recommended
3. Endpoint: auto-switches to the provider default (editable)
4. Fetch cap per session: 10 by default; further fetches are refused in that session
5. Local proxy: see next section

Proxy configuration (when the network is blocked)

Some network environments block direct connections to third-party services (e.g. r.jina.ai DNS-poisoned); a local HTTP proxy is needed. Enter your proxy address in the "Local proxy" field (example port — use your actual local proxy port):

http://127.0.0.1:27822

• Takes effect per request (undici ProxyAgent dispatcher), without affecting other traffic
• Blank = direct connection

Fetch cache and target safety

• Fetch cache (card level-2): toggle + TTL (seconds). Repeated fetches of the same URL within the TTL are served from memory — no third-party quota, no budget. Enabled by default, 600s TTL.
• Content cap (card level-2 "Max content chars"): max characters of one fetched body; longer content is truncated and marked. 0 = no cap (default 100,000, the contract-v1 bound).
• Reject private targets (card level-2): refuse loopback / private / reserved targets before forwarding, preventing the model from reaching internal resources. Enabled by default.
• Tool name (config toolName, not in the card): web_fetch_url (default) / web_fetch / auto. Choosing the official name web_fetch auto-falls-back to web_fetch_url if it is taken.
• Fallback chain (card level-2): comma-separated ordered provider names (empty = adapter + fallback). Failing or quota-exhausted providers cool down (quota 3600s / normal 60s with exponential backoff) and recover automatically; the live order and cooldown state are shown below.
• Structured output: web_fetch_url returns { url, statusCode, content, title?, headings[], links[], wordCount, readingTimeSec }.

Self-hosted fetch tool (Crawl4AI, zero API key)

When you do not want to depend on commercial providers, self-host Crawl4AI (Docker) plus a contract-v1 wrapper, registered as a custom provider — no API key needed, still "courier mode".

Auto-managed (built in, no commands): after dsh web starts, whenever the primary or fallback provider is a loopback custom provider (e.g. crawl4ai → http://127.0.0.1:8787), the plugin automatically starts the wrapper process and best-effort ensures the Crawl4AI container is running. Requirements: Docker Desktop is running (set it to start on login) and the image is pulled.

Manual fallback / first image pull:

powershell -ExecutionPolicy Bypass -File scripts/start-crawl4ai.ps1

Register in the card: Custom providers → name crawl4ai / type custom (contract v1) / endpoint http://127.0.0.1:8787 → save → select it in the "Default provider" dropdown.

Contract v1 spec: docs/contract.md; selection and verification: docs/custom-crawler-research.md.

Verification scripts

node scripts/verify-tavily.mjs            # requires TAVILY_API_KEY
node scripts/verify-jina.mjs              # anonymous OK; pass a key as argv or set JINA_API_KEY
node scripts/verify-firecrawl.mjs         # requires FIRECRAWL_API_KEY

Development and building

git clone https://github.com/tallahandsome-ux/dsh-fetch-third-party.git
cd dsh-fetch-third-party
pnpm install        # build tools + type deps (@deepseek-ai/* declared as peer/dev, from npm)
pnpm build          # tsdown dual output → lib/index.js (Host) + lib/client.js (browser card)
pnpm typecheck      # tsc --noEmit
pnpm test           # vitest (settings resolution / budget / B1 regression)

• Runtime: @deepseek-ai/* is provided by the dsh host (resolved from the profile node_modules after install); the build keeps them external, and the only real runtime dependency of the plugin is undici.
• Local dev: after changing src/, run pnpm build to rebuild lib/, then restart dsh web.

Project boundaries and third-party attribution

Original to this plugin (MIT)

• dsh-fetch-third-party itself: provider registration, primary+fallback routing, per-session budget, settings bridge, GUI card, contract-v1 custom adapter, local-stack auto management (local-stack + watchdog)
• Fetch contract v1: the integration standard for custom fetch services (see docs/contract.md)

Third-party projects used

Boundaries

• The plugin never bundles or embeds any third-party crawler engine. Crawl4AI, Jina Reader, etc. are deployed by the user on demand; the plugin integrates them via contract v1 or by reusing a built-in adapter, and the target URL is fetched only by the configured third-party service.
• API keys live only in the managed credentials store — never in code / config / docs; the plugin reads them per request and the card never echoes them.
• Keys are independent of the install method: however you install (git / local / npm), the plugin only reads keys from the local credentials store (%USERPROFILE%\.dsh\.credentials.yaml, 0600) by reference name — the repo, code, and docs contain no secrets, and cloning from GitHub carries or leaks nothing.
• The only outbound surface of the local machine is the configured third-party endpoint (the cost of the courier model; the trust boundary is yours to choose).
• Self-hosted Crawl4AI has its own SSRF protection, complementing the "never connect directly" safety premise of this plugin.

License

MIT License — see LICENSE