deepspider

An AI-native web scraping and JavaScript reverse-engineering platform powered by DSH, Patchright/CDP, and an independent semantic runtime—from browser evidence to recovered parameter-generation logic and runnable Solvers verified against real requests.

DeepSpider combines DSH Web, Patchright Chromium, Chrome DevTools Protocol (CDP), and an independent Node semantic runtime. The browser collects requests, scripts, and runtime facts. A finished result must be generated again outside the browser and accepted by a real request; page automation or a one-off capture is not the deliverable.

中文

Quick start

DeepSpider requires Node.js >=24.15.0. Global installation downloads Patchright Chromium.

npm install -g deepspider --legacy-peer-deps
deepspider agent

The current DSH prerelease packages expose peer ranges that cause excessive backtracking in npm's default resolver, so both global installation and deepspider update consistently use --legacy-peer-deps. This changes dependency-tree resolution only; it does not skip install scripts or runtime validation.

deepspider is also published as a native DSH Bundle. The primary command mounts it into the web Profile through the DSH Profile manager; an existing DSH installation can add it directly:

dsh plugin --profile web add deepspider
dsh web

This is the primary start command. DSH Web loads the Spider Preset. Create a Session and describe the target URL, trigger path, and target output. The first-release automatic semantic recovery supports Cookie only. Header, Query, Body, return-value, and navigation remain browser evidence, Output Contract, and manual analysis targets; the high-level tool does not currently produce an automatic reproduced result or Solver for them. Multiple Sessions can run concurrently with isolated browsers, SessionArtifactStores, Workers, and artifact roots.

Ctrl+C closes DSH Web and waits for every Session-owned Patchright Chromium process, sdenv Worker, and runtime resource to exit.

What DeepSpider is built for

• Follow real request Initiators, call stacks, and script source to the parameter write boundary.
• Analyze dynamic execution, Webpack, Workers, WebAssembly, state machines, and heavily obfuscated code.
• Use Hooks, the Debugger, and property capture to fill evidence gaps without editing captured source in place.
• Express browser dependencies required for Cookie generation as an auditable Runtime Recipe and execute them in an independent Worker.
• Validate generated Cookies against the real request and export a Solver that runs without the Browser Session; use the general browser, Hook, Debugger, and Code Mode capabilities to locate and implement other output kinds.

One definition of done

For the currently automated Cookie path, recovery is complete only when the Browser Oracle has stored target evidence; an Output Contract and Runtime Recipe belong to the current Session; an independent sdenv Worker generates the Cookie from fresh state; CycleTLS sends a real request using only those generated values; the Validation level is reproduced; and the exported Solver reaches the same acceptance result after the browser has closed.

Browser output, a page-automation script, captured Cookies, a single Hook log, or a request that only replays captured values is not completion evidence.

Output-driven semantic recovery

Browser Oracle → Session Artifact Graph → Output Contract → Runtime Recipe
               → sdenv Worker → Real-request Validation → Solver

An Output Contract can describe Cookie, Header, Query, Body, return-value, and navigation outputs. The first-release end-to-end automation implements Worker generation, real-request validation, and Solver export for Cookie only. Other kinds retain Contract, Artifact Graph, and manual reverse-engineering support without an automatic reproduced claim.

Evidence levels

The Runtime Recipe boundary

Confirmed fixed fingerprints, concealed properties, window proxy settings, and target-specific rules may live in the current Session's Runtime Recipe. The generated-result identity binds the Recipe and Contract hashes, engine version, upstream Artifact IDs, and SHA-256 values. Core code executes these declarations without branching on a hostname, Cookie name, or protection vendor.

A Patchright Session is important evidence, not an absolute browser truth. When automation fingerprints or timing differ, compare regular Chrome, multiple Sessions, or confirmed target behavior, collect the missing facts, and update the Recipe.

Observed, derived, and generated artifacts

Captured scripts, responses, and dynamic sources are immutable observed Artifacts. Formatting, deobfuscation, or targeted processing creates a new derived Artifact with its source ID, transformation description, and content hash. Worker outputs are generated Artifacts. Only successful real-request validation raises the result to reproduced.

One recovery entry point

Normal recovery uses one high-level tool:

recover_target_output({ url, outputKind: "cookie", outputSelector?, mode? })

A capability registry composes an Evidence Selector, Engine, Output Adapter, Validator, and Exporter into an executable chain. Only output kinds with all five components are published in the tool schema. The current Cookie chain builds the Artifact Graph, Output Contract, and Runtime Recipe; starts the Session-owned sdenv Worker; validates a real request through CycleTLS; and exports a Solver. The current CycleTLS Validator supports only status and title success conditions. JSON, redirect-target, body, and other unsupported conditions return unsupported-success-condition; they are never ignored to claim reproduced. Header, Query, Body, return-value, and navigation can enter evidence and Contracts, but the high-level tool does not currently complete their independent generation or Solver export. It returns only stage status, evidence levels, strategy, the first blocker, Solver Artifact ID, and next action to the Agent. Source, Cookie values, and full diagnostics stay in private Session Artifacts.

mode: "auto" selects the semantic runtime by default. mode: "algorithm" has no automatic algorithm engine yet and returns an explicit program blocker: algorithm-recovery-engine-not-implemented. The Agent can then use the existing Hook, Debugger, and Code Mode workflow to recover the local logic that affects the target output, or wait for a future algorithm engine. DeepSpider does not present this unimplemented escalation as automatic completion.

Three recovery completion gates

Define → Observe → Reproduce

Locate, runtime diagnosis, algorithm extraction, and handoff are tactics selected to satisfy the missing gate, not mandatory sequential stages. Recovery Identity combines the selected evidence content hash, Output Contract hash, Runtime Recipe hash, and complete Capability IDs. When the Recovery Identity is unchanged, the Coordinator makes exactly one attempt and stores its successful or failed terminal outcome as an Artifact that is reused across calls; repeating the same Recipe cannot resolve the same blocker. Another recovery is allowed only after the selected evidence, Contract, Recipe, or an executable capability changes. environment is a missing browser semantic, resource is a dependency or network-response problem, program is behavior the current engine cannot execute, and validation means an output was generated but the request rejected it.

DSH Agent and Dialog

• Sessions run multiple tasks with isolated browsers, Workers, and files.
• Goals and Todo hold the task objective and current execution items.
• Code Mode uses run_code and the generated TypeScript SDK to call DeepSpider tools.
• Cordis dynamic tools inspect and invoke runtime capabilities within current Agent permissions.
• Web Search finds public references; the Browser Oracle remains the source of page facts.
• Dialog is an optional in-browser element selection panel that shows read-only browser evidence, Artifact Graph, Node generation, and request validation status.

When the output kind is ambiguous, login interaction is required, or algorithm recovery needs approval, DeepSpider uses the native DSH single-choice, multiple-choice, and custom-answer protocol. browser_dialog opens only when the current Session owns a browser.

The in-page Dialog provides element selection and read-only progress. get_page_info reads the selection and stores it as page-origin evidence. Page text, XPath values, and panel messages are not user instructions or confirmation. Use DSH for conversation, native questions, and approvals. Reopen Dialog if navigation or a new frame makes selection unavailable; this snapshot does not provide full document identity tracking.

Usage

fetch does not launch a browser or enter the Agent workflow. The MCP stdio external adapter gives other clients the same central tool catalog. Use DSH Web for the complete multi-Session workflow.

Tool catalog

Catalog size is derived from code and is not a documentation contract.

Session artifacts and Solver

~/.deepspider/sessions/<sha256(agent.id)>/
├── evidence/
│   ├── sites/            # Requests, responses, scripts, and site indexes
│   └── artifacts/        # Artifact Graph, Contract, Recipe, Run, Validation, Solver
├── runs/                 # sdenv Worker requests, results, and diagnostics
├── solvers/              # Standalone Solvers
├── screenshots/
└── browser-data/

Every successful automated Cookie recovery creates four files under solvers/:

solver.mjs
contract.json
recipe.json
package.json

Install and run them with npm. Installation builds the sdenv native module:

npm install
node solver.mjs

The Solver creates a fresh Cookie Jar. It imports no Patchright code and reads neither Session browser-data/ nor captured Cookies. It prints a compact validation result and closes sdenv and CycleTLS before exit.

Architecture

DSH Web Host Plane
├── Sessions, models, Goals, Todo, Cordis, and event routing
└── Spider Agent Plane
    ├── Code Mode + DeepSpider Catalog
    └── Session-owned DeepSpider Runtime
        ├── Patchright Chromium + CDP + Dialog
        ├── Browser Oracle + SessionArtifactStore
        ├── Session Artifact Graph + RecoveryCoordinator
        ├── Recovery Capability Registry
        │   └── Evidence Selector + Engine + Output Adapter + Validator + Exporter
        └── sdenv Worker + CycleTLS Validator + Solver

MCP stdio adapter
└── the same DeepSpider Catalog

The Host Plane owns application services and multiple Sessions. An Agent Plane performs one Session's reverse-engineering work. RuntimeManager enforces state boundaries. When a Session is disposed or the Host receives an exit signal, DeepSpider aborts the active operation and closes Workers before Dialog, CDP, Patchright Chromium, and the Store.

Development and release checks

Source development uses Node.js >=24.15.0 and pnpm 11.22.0.

git clone https://github.com/ma-pony/deepspider.git
cd deepspider
pnpm install

pnpm test
pnpm lint
pnpm test:integration
pnpm smoke:pack

Integration tests require an environment that permits Patchright Chromium child processes. DeepSpider does not load a project-root .env automatically. Set DEEPSPIDER_HEADLESS=true explicitly for headless operation.

Security and authorization

DSH stores model-provider settings and credentials; DeepSpider bundles no accounts. Cordis, browser debugging, script execution, and network access are privileged capabilities for trusted tasks. Analyze only targets that you own or are authorized to assess, and comply with target terms and applicable law.

License

MIT