dsh-ros2

ROS2 debugging toolset and robot-state vision analysis for DeepSeek Harness (DSH), shipped as a plugin. 中文版见 README_CN.md.

版本对应关系：npm 上的 dsh-ros2@0.1.0 就是本仓库当前版本（monorepo 布局）。 GitHub 的 v0.8.0 ~ v0.15.0 标签是已废弃的旧单体布局历史，从未发布到 npm。 版本号在 2026-08 monorepo 拆分时重新基线。详见 docs/versioning.md。

dsh-ros2 gives a DSH agent full robot development / debugging capabilities on any host with ROS2, organized in four capability tiers:

All tools run plain ros2 / colcon / rosdep CLI commands on the host; L1 never modifies anything, L2 always asks first.

Screenshots

RViz2 offscreen render (latest lite_urdf, real material colors)	Head camera	Wrist-left camera	Wrist-right camera

Left: rviz_offscreen_node renders with the real rviz stack (OGRE) and publishes to the /rviz/scene image topic. Right: three frames grabbed from live camera topics by ros2_image_snapshot (1280×720). Full test record: docs/test-robot-state-vision.md.

Features

• Zero-intrusion diagnostics: 83 tools cover most ROS2 debugging scenarios — from "is the package installed?" to "what is on this topic right now?", one command, one answer;
• Whole-graph topology: ros2_graph folds nodes/publishers/subscribers/services/actions into one JSON — see the system structure in seconds;
• Approval-gated writes: builds, dependency installs, message scaffolding etc. go through the DSH approval service; fail-closed, denial = failure;
• Visualization as a service: "see" headlessly — screenshots / multimodal description / window interaction are fully local, no remote display;
• Parallel realtime vision: the VLM runs in a separate ROS2 process (vlm_node, service /vlm/describe); images come from topics (sensor_msgs/Image / CompressedImage); vision_bringup auto-creates one bridge per image topic, headless-ready;
• RViz2 offscreen rendering (motion render ~22 Hz on llvmpipe; 30 Hz full rate with GPU): the real rviz render kernel (rviz_common + OGRE) renders any .rviz scene on a virtual display and publishes it as an image topic — no screenshots, no X11 window-stacking dependency. Performance optimizations: open3d low-poly meshes (scripts/simplify_visual_meshes.py) + direct OGRE pixel read (no PNG round-trip) + double-render elimination → motion rendering 1.9 → ~22 Hz on llvmpipe (11×), and 30 Hz full rate with NVIDIA GPU passthrough (v0.9.3), memory −2.5×;
• Real-time safety framework (safety_monitor node + robot_safety_* tools, see Safety framework): layered defense — tool-layer safety gate (pre-execution /safety/state check) → reactive monitors (motion tracking/stall with hysteresis, joint-feedback loss, watchdog, optional torque) → event-driven VLM semantic arbitration → human arbitration. Latched NORMAL/LOCKED state machine (lock persists until a human unlocks; non-fatal events never lock); every threshold/topic/lock-action is registered per robot in the profile safety section; the geometric pre-check (joint limits / velocity / FK self-collision) is a reserved layer;
• Bundled skills (9): one carrier per recurring journey — ros2-diagnostics (which tool when, narrow-down methodology), ros2-bringup-recovery (environment → launch → verify), ros2-liveness-triage (rates and payload in one or two calls), ros2-tf-integrity (edges, lookups, missing broadcaster), robot-registration / robot-retrieval (body profile and instant reuse), robot-state-vision-analysis (status → render → VLM → cross-check), robot-motion-control (the one plan → validate → approve → execute → verify path) and robot-safety-procedure (latch, LOCKED, boundaries).

Quick start

Requirements

• A host with ROS2 (Jazzy verified; Humble should work), ros2 on PATH;
• Node ^22.19 || >=24 (DSH host requirement);
• L4 vision additionally needs Python 3 rclpy and an OpenAI-compatible VLM gateway (e.g. Gemini or a self-hosted gateway).

Install the plugins (9 npm packages, since the monorepo split)

Install the domain bundles you need — or the dsh-ros2 aggregate, which mounts core/profile/moveit/safety/vision (81 tools + the 9 skills). Add the optional dsh-ros2-state for the family's full 83 tools. All packages are published to npm at 0.1.0 (see docs/versioning.md for the GitHub ↔ npm version correspondence).

Via the DSH plugin CLI (recommended; resolves through npm):

# full set (aggregate; pulls core/profile/moveit/safety/vision as deps)
dsh plugin --profile <profile> add dsh-ros2
# or lean installs — e.g. diagnostics only:
dsh plugin --profile <profile> add dsh-ros2-core
# dsh-ros2-common is a plain library, auto-installed as a dependency.

Via npm directly (same packages; useful for inspecting/installing the optional add-ons):

# the aggregate (mounts dsh-ros2-common/core/profile/moveit/safety/vision = 81 tools)
npm install dsh-ros2
# optional add-ons — the sidecar data plane + its control-plane state client.
# `dsh-ros2-state` is NOT part of the aggregate's patch; its own 2 tools
# (state_get / state_snapshot) are what take the family from 81 to 83:
npm install dsh-ros2-state dsh-ros2-sidecar

Package roles: dsh-ros2 (aggregate) · dsh-ros2-common (shared library) · dsh-ros2-core (core diagnostics) · dsh-ros2-profile (profiles/topology) · dsh-ros2-moveit (MoveIt motion) · dsh-ros2-safety (safety framework) · dsh-ros2-vision (vision pipeline) · dsh-ros2-state (state client) · dsh-ros2-sidecar (data-plane daemon).

Prompt guidance. dsh-ros2-core contributes a system-prompt section that points the model at this toolchain for ROS2 work — discover the graph with ros2_*, load the body with robot_load/robot_topology, plan and act through moveit_*/robot_safety_* (approval-gated, gated on /safety/state) — instead of ad-hoc ros2 CLI calls or private rclpy scripts. Any install that includes core therefore carries it, the dsh-ros2 aggregate included.

The section is rebuilt on every assembly from the tools that are registered at that moment, so a core-only install is never told to call moveit_move or robot_safety_* it does not ship. It is registered with the plugin's own lifetime: disabling or removing the plugin removes it from the prompt, so an unloaded dsh-ros2 leaves no residue.

Minimal configuration (per-bundle, whole-object replacement)

After the split, each bundle carries its own run-seam config (same keys repeated per id) and the vision provider lives only on dsh-ros2-vision:

# fragment of a DSH profile patch (id-targeted config overrides)
- id: dsh-ros2-core                 # also: -profile / -moveit / -safety
  config:
    rosSetup: source /opt/ros/jazzy/setup.bash &&   # prepare the ROS2 environment
    workspaceRoot: /home/you/ros2_ws                 # default cwd for colcon/rosdep
- id: dsh-ros2-vision
  config:
    rosSetup: source /opt/ros/jazzy/setup.bash &&
    vision:
      provider: gemini                               # mock | gemini | openai
      apiKey: ${VLM_API_KEY}                         # ${ENV} 引用从环境变量解析，勿明文写 key
- id: dsh-ros2-safety
  config:
    safetyStrict: warn                               # 'warn' (default) | 'reject' (fail-closed); LOCKED always rejects

Three-minute taste

ros2_topology --tf                  # WHOLE system (nodes/topics/services/actions/TF) in one call
ros2_graph                          # whole system topology in one shot
ros2_topic_list                     # all topics and types
ros2_topic_echo /joint_states       # sample one frame of joint states
ros2_tf_list                        # TF tree edges
ros2_doctor                         # system health report

Tool reference

Tools live in domain packages: L1/L2/L3 → dsh-ros2-core; motion_validate/moveit_* → dsh-ros2-moveit; robot_*/ros2_zero_pose_semantics → dsh-ros2-profile; robot_safety_* → dsh-ros2-safety; vision tools → dsh-ros2-vision.

L1 read-only diagnostics

L2 management (approval-gated)

Every L2 tool performs a write operation and asks the user first via the DSH approval service (fail-closed when unavailable or denied). The read-only helpers ros2_jobs_list / ros2_job_status need no approval.

L3 visualization

GUI lifecycle + screenshots + multimodal vision ("see first, then move") + xdotool-level interaction ("see and move"). Screenshots use Pillow ImageGrab on X11 (no extra CLI install); vision providers are pluggable; interaction requires xdotool (sudo apt install xdotool).

Interaction recipes (model-facing): orbit the RViz2 view with ros2_gui_interact {action: "drag", windowTitle: "rviz2", button: 1, toX: <dx>, toY: <dy>}, zoom with action: "drag", button: 3, reload a display config with ros2_gui_interact {action: "key", keys: "ctrl+shift+r"}. When wmctrl cannot enumerate windows (e.g. no window manager on the display), window-relative interaction reports "window not found" — fall back to absolute screen coordinates, which now asks for approval because it targets whatever window is focused.

Why the allowlist: action=key types arbitrary text into the targeted window, so an unrestricted target is a host-level input-injection path (typing into a terminal runs commands). The allowlist is a convenience control — any process can retitle its own window, so a title match is not proof of identity; the approval gate is the boundary, and it fails closed (no approval service = denial). See docs/safety.md.

L4 realtime vision (parallel VLM over ROS2, headless image topics)

Perception matches the robot-control stack: the VLM runs in a separate ROS2 process (vlm_node, service /vlm/describe + cached topic /vlm/description), and images come from sensor_msgs/Image topics, never X11 screenshots — headless-ready. Acquisition (ros2_image_snapshot) needs no custom package (plain rclpy); only ros2_vlm_analyze/ros2_vision_analyze require the dsh_ros2_vlm ROS2 package (vlm/) — build/run see docs/architecture.md §4. When the pipeline is down, ros2_vlm_analyze/ros2_vision_analyze return an explicit VLM_UNAVAILABLE + degradation hint, and the snapshot JPEG can be read directly by the Agent's own multimodal model.

# build + launch the vision pipeline (auto bridge per image topic)
# Build the workspace OUTSIDE /tmp: a reboot wipes /tmp, and a `rosSetup` that
# sources a deleted workspace is the misconfiguration documented in
# docs/feedback-env-recovery.md. Use any persistent path (`~/vlm_ws` here).
mkdir -p ~/vlm_ws/src && ln -s <repo>/vlm ~/vlm_ws/src/dsh_ros2_vlm
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash
VLM_API_KEY=... ros2 run dsh_ros2_vlm vlm_node &       # parallel VLM process
ros2 run dsh_ros2_vlm vision_bringup &                 # discover topics, one bridge each

RViz2 offscreen rendering (dsh_ros2_rviz_offscreen)

The real rviz rendering stack (rviz_common + OGRE + rviz_default_plugins) loads a .rviz scene offscreen under Xvfb and publishes it to the /rviz/scene image topic — read from the render kernel, not X screenshots, no window-stacking dependency.

# build (needs a colcon workspace like vlm_ws — keep it out of /tmp)
ln -s <repo>/offscreen ~/vlm_ws/src/dsh_ros2_rviz_offscreen
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash
# run (config_path points at a .rviz scene file)
xvfb-run -a -s "-screen 0 1280x800x24" ros2 run dsh_ros2_rviz_offscreen rviz_offscreen_node \
  --ros-args -p config_path:=/tmp/robot_scene.rviz -p topic:=/rviz/scene \
  -p width:=800 -p height:=600 -p rate:=5.0

Robot body mesh rendering essentials (pitfalls collected; details in docs/architecture.md §4.4):

1. Jazzy RobotModel properties: use Description Source: Topic + Description Topic: <topic> (the legacy Robot Description: is ignored → Links empty);
2. mesh paths: use absolute paths or a file:// prefix in the URDF (bare paths fail in resource_retriever); the description publisher must stay alive (transient-local, late subscribers miss it otherwise);
3. URDF must bind to TF by name: URDF link names must exactly match the live TF frame names (publishing the robot's actual /robot_description works). Mismatch → every link transform lookup fails → all meshes pile at the fixed-frame origin;
4. View distance: Orbit Distance ≈ 1.5–2.0 m for an RViz-like close full-body view (> 5 m shrinks the robot to a tiny center blob);
5. Health signal: ~3 s after startup the node logs FM: ... frames=N and transformHasProblems(...)=0 — meshes correctly bound to TF.

MoveIt2 motion — five essential modes, one tool

Design intent. Robot motion through MoveIt is, at its core, only five operations: set joints absolutely, nudge joints relatively, place the end-effector at an absolute pose, move it by a relative delta, or execute a pre-planned trajectory. Rather than a growing zoo of named tools, dsh-ros2 abstracts them into one tool moveit_move with a mode parameter — so the interface stays small, predictable and scriptable, and works with any MoveIt package (it reads the SRDF and speaks only standard moveit_msgs).

moveit_move {mode: "joint_abs", group: "right_arm", joints: "right_shoulder_roll:=0.5"}
moveit_move {mode: "joint_rel", group: "right_arm", deltaJoints: "right_elbow_pitch:=-0.2"}
moveit_move {mode: "pose_rel",  group: "right_arm", deltaPose: "0.05 0 0 0 0 0"}
moveit_move {mode: "pose_abs",  group: "right_arm", pose: "0.5 0 0.8 0 0 0"}
moveit_move {mode: "trajectory", group: "right_arm", trajectory: "/tmp/traj.json"}

How it works. moveit_discover/moveit_status tell you what you can move (groups, joints, EE link from the SRDF chain tip, online state); moveit_move turns any mode into a standard MoveGroup goal (/move_action), executes via /execute_trajectory, and — with planOnly + trajectoryOut — saves the planned trajectory so mode: "trajectory" can run it later (plan/execute separation). Nothing is bound to a specific MoveIt package; only the SRDF path matters (auto-resolved by package scan, or explicit srdf/package).

Safety framework

Design intent. A two-tier strategy that keeps real-time and semantic judgment apart (full contract: docs/safety-handover.md, the handover doc for downstream robot-adaptation agents — generic framework/interfaces belong here, body-specific data sources/schemes/algorithms belong downstream):

tool-layer safety gate (pre-execution: /safety/state LOCKED; monitor-down per safetyStrict)
  → reactive monitors (in-execution: motion tracking/stall + hysteresis,
    joint-feedback loss, watchdog, optional torque; detect at control frequency,
    response budget ≤100 ms)
  → VLM semantic arbitration (post plan-change / anomaly, seconds — pulled up only when needed)
  → human arbitration (non-safe verdicts always escalate; human-gated unlock)

• safety_monitor node (dsh_ros2_safety package): subscribes the joint feedback (+ optional commanded stream / torque stream), runs the checkers on a control_frequency timer, and latches LOCKED on any CRITICAL event — the latch persists until a human unlocks (no auto-reset into the same danger). Publishes /safety/state (transient-local), /safety/event, /safety/heartbeat, /safety/lock_active; services /safety/get_state, /safety/unlock, /safety/set_lock.
• Latched lock, non-fatal never locks: any CRITICAL event latches LOCKED (the latch persists until a human unlocks — no auto-reset into the same danger); watchdog separates critical (down → lock) from observed (down → WARNING only); single-frame noise is filtered by M-of-K hysteresis; a WARNING never latches. Tool-layer fail-closed on monitor-down is per safetyStrict: 'reject' (default 'warn' proceeds with a warning).
• Per-robot registration: robot_register writes a generic safety section (URDF-derived velocity/effort limits) and auto-launches the monitor; robot_profile.py safety set <key> <json> updates any threshold/topic/list (schema-validated). Torque checking auto-disables when no effort feedback exists. Reserved layers (interfaces registered, not implemented): the geometric pre-check (joint-limit / velocity / FK self-collision on the command path — motion.max_velocity / max_acceleration), the computed-torque feedforward input (torque.feedforward_topic), YOLO-style lightweight triggers, and the non-ROS estop path (estop).
• Tool-layer gate: moveit_move consults /safety/state before executing — LOCKED always rejected (SAFETY_LOCKED); if the monitor is unreachable, safetyStrict: 'reject' (fail-closed) or 'warn' (default) proceed-with- warning. robot_safety_arbitrate runs the fixed-format VLM arbitration and flags any non-safe verdict for human arbitration.
• Forensics: a ring buffer of joint/torque samples is dumped to forensics.dump_dir on every CRITICAL lock, for post-hoc / VLM diagnosis.

# build the safety package (same colcon workspace as vlm/ and offscreen/)
ln -s <repo>/safety ~/vlm_ws/src/dsh_ros2_safety
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash

The safety_core pure logic ships with a fault-injection self test (python3 packages/safety/safety/scripts/safety_core.py --selftest, 12 scenarios) — no ROS2 needed to verify the state machine.

Robot profiles & communication topology

Design intent. A robot's body (URDF links/joints, cameras, MoveIt groups, zero-pose semantics) and its comms graph are costly to rediscover every time. The plugin persists them as a structured profile (~/.dsh-ros2/robots/<name>.yaml): register once on first contact, then load instantly forever after. For the comms graph, full verbosity does not scale (a complex robot has hundreds of topics/services), so the design is a trade-off: an aggregate snapshot (light node/topic/service lists) plus progressive learning of only the important nodes as you actually work with them.

Two bundled skills complete the workflow: robot-registration (first-contact flow: ask name/URDF → collect → register → verify, plus the topology baseline snapshot) and robot-retrieval (load the profile instantly and bring up renders/diagnostics/motion from it, including the learned topology and zero-pose semantics).

How it works. Everything is plain structured YAML under ~/.dsh-ros2/ — robots/<name>.yaml (body + topology) and zero-pose.yaml (calibration). robot_register snapshots the body; robot_topology snapshot snapshots the aggregate layer; robot_topology learn appends one node at a time (idempotent merge) as you discover what matters. robot_load and robot_topology show read them back instantly — one call instead of N discovery calls.

The knowledge base is consumed, not just stored. robot_topology diagnose (L1, read-only) is the entry point for knowledge-augmented diagnosis: it loads the learned nodes + snapshot and cross-references them against the LIVE ros2 graph —

• missing: learned nodes that are offline now (controllers/publishers down) — highest priority;
• new: live nodes not in the knowledge base (learn candidates);
• matched[].drift: per learned node, expected pub/sub/srv/act vs actual (a missing topic = a gone connection; a new topic = the node changed);
• topic_drift: aggregate snapshot topics vs live topics.

The ros2-diagnostics and robot-retrieval skills both start diagnosis with diagnose and close the loop by learning important new nodes — so the knowledge base improves with every session and every diagnosis gets faster.

Bundled skills

Configuration

rosLogDir also covers ROS2 Python CLIs spawned by tools (topic echo/pub, ros2 run); additionally runCommand auto-falls back to a writable dir when ~/.ros/log is not writable.

Project layout

dsh-ros2/                      # pnpm monorepo (workspace root, private)
├── pnpm-workspace.yaml        # packages/*
├── tsconfig.base.json
├── packages/
│   ├── common/                # dsh-ros2-common (not a bundle): runner / parse / toolkit + scripts/robot_profile.py (zero-copy)
│   ├── core/                  # dsh-ros2-core (61 tools): L1 diagnostics + L2 management + L3 GUI + diagnostics/bring-up/liveness/TF skills + gui.ts + pty_session.py
│   ├── profile/               # dsh-ros2-profile (4 tools): robot_register/load/topology + zero-pose calibration + registration/retrieval skills
│   ├── moveit/                # dsh-ros2-moveit (4 tools): discover/status/motion_validate/moveit_move + moveit_*.py + motion_validator.py + robot-motion-control skill
│   ├── safety/                # dsh-ros2-safety (5 tools): robot_safety_* + safety/ ROS2 pkg + safetyStrict config + robot-safety-procedure skill
│   ├── vision/                # dsh-ros2-vision (7 tools): vision tools + vlm/ + offscreen/ ROS2 pkgs + vision provider service + state-vision skill
│   └── dsh-ros2/              # aggregate bundle (empty apply, backward compat)
├── docs/                      # architecture.md · safety.md / safety-handover.md / safety-todo.md / safety-gpt-review.md · test-*.md · plugin-split-plan.md
├── .github/workflows/         # CI: Node 22/24 → workspace typecheck/test/build + per-package tarball validation
└── CHANGELOG.md               # version history (Keep a Changelog)

Plugin split (9 packages)

Since v0.15.0 the plugin is a pnpm monorepo of 9 npm packages (per docs/plugin-split-plan.md, ISP-tightened): 83 tools + 9 skills (the four original skills preserved with unchanged names and behavior). Install the domain bundles you need (or the dsh-ros2 aggregate for the full set):

• dsh-ros2-common is a plain library (not a cordis bundle) — shared runner, parsers, toolkit and scripts/robot_profile.py (zero-copy).
• Cross-package runtime contracts stay unchanged: /vlm/describe, /safety/state, /safety/set_lock …; safetyStrict semantics unchanged.
• dsh-ros2-vision fixes the npm publish defect: its files include vlm/ + offscreen/.
• The vision provider is an optional cordis service (dshRos2.vision); ros2_gui_observe (core) and ros2_zero_pose_semantics (profile) degrade to VISION_UNAVAILABLE when it is absent.

Troubleshooting / FAQ

• ~/.ros/log Permission denied: ROS2 cannot write its log dir. Set rosLogDir (e.g. /tmp/ros-log); runCommand also falls back automatically.
• A flood of RTPS_TRANSPORT_SHM / FastDDS SHM warnings in stderr: harmless noise when SHM transport is unavailable (common in containers/restricted environments); tools drop it by default.
• ros2 topic echo returns empty: check ros2_topic_info -v for publisher count and QoS; sample transient-local topics with --qos-durability transient_local.
• Offscreen render: "all parts piled at origin": URDF link names do not match TF frame names (see mesh essentials #3); check the node log FM transformHasProblems(<link>)=1 first.
• RobotModel shows no meshes (Links empty): on Jazzy you must use Description Source/Topic; the legacy Robot Description: does nothing.
• Could not load resource ... Unable to open file: mesh paths need absolute paths or a file:// prefix.
• Description missing after the publisher exits: the URDF publisher must stay alive (transient-local only re-sends once to late subscribers; gone when the process exits).
• vision_bringup missed some image topics (2/4 measured): fixed — it now refreshes discovery every --refresh seconds (default 10), auto-spawning bridges for topics that appear later and stopping bridges for topics that disappear.

Development

Install with **pnpm 11.x** (the repo pins `packageManager: pnpm@11.22.0`; CI installs the matching
pnpm via `pnpm/action-setup@v6`). Node `^22.19 || >=24` is required.

```bash
pnpm install
pnpm run typecheck   # tsc --noEmit
pnpm run test        # vitest (425 cases; plus 11 sidecar + 15 zero-pose + 38 robot-profile + 16 private-path Python checks)
pnpm run build       # tsc -> lib/ + lib/types/
pnpm run audit       # supply-chain gate: runtime graph, high+ (the exact command CI runs)
pnpm run audit:all   # ... plus the dev-only tooling that CI's gate deliberately excludes

CI (.github/workflows/ci.yml): on push to main / PRs, runs typecheck/test/build on Node 22 and 24, and validates that the pnpm pack artifact contains the patch layer (cordis.patch.yml) and the build output.

Release workflow (npm & GitHub Releases): see PUBLISH.md.

Roadmap

• vision_bringup polling/refresh discovery (auto-bridge late topics, stop bridges for gone topics);
• Zero-pose semantics: generic calibration flow (ros2_zero_pose_semantics, render + VLM + user confirm, 3-axis combos) linked into robot profiles;
• npm publishing (9 packages @ 0.1.0 on npmjs, published 2026-08-30; dsh-ros2-state/dsh-ros2-sidecar added 2026-09; see docs/versioning.md);
• More ROS2 distros (Humble / Rolling) compatibility validation.

Docs

Contributing

Issues and PRs welcome (in Chinese or English). Please keep pnpm run typecheck && pnpm run test && pnpm run build green and update the relevant docs.

Acknowledgments

• DeepSeek Harness — plugin host framework;
• ROS2 / RViz2 community — rendering and toolchain foundations.

License

MIT