Skip to content

Add the ARX ONE real-robot integration on the shared tools - #35

Merged
1rubbishyuan merged 6 commits into
Open-RUA:mainfrom
Cyberkona:arx-one/integration
Nov 11, 2026
Merged

1rubbishyuan merged 6 commits into
Open-RUA:mainfrom
Cyberkona:arx-one/integration

Conversation

@Cyberkona

Copy link
Copy Markdown
Contributor

Summary

This PR adds ARX ONE (two ARX X5 arms, three RealSense D405 cameras) as a real-robot integration, and is meant as a template for other robots. Real-robot behavior that is not specific to this robot goes into the shared layers as options. Simulation and real robots run the same tools and choose a mode:

  • Execution mode (--execution-mode): parallel (default, unchanged) or replanning. In replanning, each world-mutating result is read before the next world-mutating call. A failed action cancels the queued plan, and new actions wait for a confirmed stop and a fresh observation.
  • Motion mode (environment metadata motion_execution): stepwise (default, unchanged) or wholepath. In wholepath, a move is one preflighted path, and gripper-only actions hold every joint at its measured angle for steps control periods.
  • Environment-declared tool settings: finish statuses, whole motion profiles, measured 0-1 gripper-opening checks, and stop confirmation from receipts.

An environment that declares nothing keeps main's tool schemas, prompts and scheduling. The integration contains no model-facing tool code; it composes the shared direct tool profile.

This supersedes #28, which ported an earlier version of this work. The commits are rebuilt on the current main as independent features.

Commits

  1. feat(tools): environment-declared finish statuses, motion profiles and opening checks. finish_statuses, motion_profiles and normalized_opening_tolerance in the environment metadata.
  2. feat(execution): the replanning execution mode beside parallel.
    • ParallelToolExecutor gets small extension points.
    • execution/replanning.py adds the mode, and create_executor() selects it.
    • --execution-mode and IntegrationSpec.execution_mode choose it.
    • ToolOutput.action (ActionOutcome) reports what an action did; it is never serialized.
    • physical_hardware requires receipts to confirm a stop.
    • Job controls return cancellation receipts.
  3. feat(tools): the wholepath motion mode. PathCommand, an optional backend step_path, preflighted paths with reachable_prefix, and joint holds for gripper-only actions.
  4. feat(arx): the ARX ONE integration. Body device process, environment adapter, perception, offline tools and deployment templates. It declares replanning. Every episode confirms the arms are idle and takes the control lease; there is no read-only episode mode, and read-only work uses the offline capture and calibration tools.
  5. build(arx): packaging. pyproject.toml and MANIFEST.in include the native sources, release_contents.py accepts them, and sdk_release.py validates the arx_one config.

Outside the integration (main code)

19 code files, plus packaging and docs. ✎ modified, + added.

uni_robot_agent/
├── cli.py                       ✎ --execution-mode {parallel,replanning}
├── config.py                    ✎ execution_mode
├── session.py                   ✎ create the executor for the mode
├── integrations/base.py         ✎ IntegrationSpec.execution_mode (default "parallel")
├── execution/
│   ├── __init__.py              ✎ EXECUTION_MODES, create_executor
│   ├── parallel.py              ✎ extension points; scheduling unchanged
│   ├── replanning.py            + replanning mode
│   └── scheduler.py             ✎ replan_observation / ends_episode facts, cancel_pending
├── services/environment/
│   ├── contracts.py             ✎ PathCommand
│   ├── backend.py               ✎ optional step_path
│   └── primitive.py             ✎ PathCommand dispatch
└── tools/
    ├── toolkit.py               ✎ ActionOutcome, model view kept after result transforms
    ├── jobs.py                  ✎ cancellation receipts
    ├── episode.py               ✎ finish statuses
    ├── scene/provider.py        ✎ capture_scene is a replan observation
    └── gripper/
        ├── contracts.py         ✎ motion profiles
        ├── opening.py           ✎ normalized opening tracker
        ├── motion.py            ✎ wholepath mode
        └── provider.py          ✎ mode-aware descriptions, ActionOutcome from receipts

pyproject.toml  MANIFEST.in  README.md  CHANGELOG.md
scripts/validation/release_contents.py  scripts/validation/sdk_release.py
docs/  architecture.md  development.md  integrations.md  running.md  services.md
Setting Where it is chosen Default
Execution mode --execution-mode, else IntegrationSpec.execution_mode parallel
Motion mode environment metadata motion_execution stepwise
Finish statuses environment metadata finish_statuses ["success"]
Motion profiles environment metadata motion_profiles shared standard / approach / precision
Gripper opening check gripper_opening_models (metric) or normalized_opening_tolerance (0-1) no check
Stop confirmation physical_hardware + receipt diagnostics.stop.status / execution_outcome a simulation counts as stopped

Inside the integration (uni_robot_agent/integrations/arx_one/, 172 files)

Directories follow where and when code runs. body/ is the device process, environment/ and perception/ serve an episode, and offline/ runs outside episodes.

arx_one/                          172
├── __init__.py config.py session.py prompt.py operator_cli.py README.md DEPLOYMENT.md
├── environment/                    9   episode over the Body RPC: admission, control lease, step_path, observations
├── perception/                    13   snapshots, RGB-D geometry, extrinsics, depth completion
├── offline/                       17   outside episodes; never imported by an episode
│   ├── fleet/                      7   site manifest, Fleet client, fleet-wide stop confirmation
│   ├── capture/                    4   read-only observation capture and archive inspection
│   └── calibration/                5   eye-to-hand and eye-in-hand calibration and verification
├── deploy/                        11   configuration templates and launch scripts
└── body/                         114   device process
    ├── core/                      19   generic: RPC, leases, operations, quarantine, arm/camera contracts
    ├── host/                      10   generic: build, install, check and serve Bodies
    └── drivers/                   82   fake/ (minimal template), realsense/, arx_x5/ (incl. vendor ROS controllers)
  • To add a robot, reuse body/core and body/host, and copy drivers/fake/ for each device kind.
  • The robot's shape (arms, joints, cameras) comes from environment.json.
  • Values tuned for the X5 are marked "Robot-specific" in environment/adapter.py. The README section "Adapting to another robot" lists them.
  • Dependencies point one way: body/ imports no agent-side code, and an episode never imports offline/.
  • The prompt stays in the integration (prompt.py).

Notes for review

  • --execution-mode returns with parallel and replanning only; the former serial value is not accepted. It is separate from --tool-return-mode and --resource-scheduling, which still configure scheduling inside the parallel executor. An internal test that asserts the option is absent (test_runtime_init) needs updating.
  • Known issue: Hybrid under replanning. actor_act reports no ActionOutcome, so a failed actor_act counts as unknown execution with an unconfirmed stop: queued actions are cancelled and world-mutating tools stay refused for the rest of the episode, in simulation too. Only the hybrid tool profile combined with replanning is affected; direct and parallel are not, and ARX ONE uses direct. This needs a follow-up that fills ActionOutcome from the actor's execution summary.
  • Commit 4 alone lacks the packaging rules; commit 5 adds them. Every earlier commit builds and imports on its own.

Validation

  • Each commit was checked out on its own: it compiles and imports, and the relevant offline tests pass. This repository ignores tests/, so the tests come from our internal suite. Commits 4 and 5 each pass 1064 ARX ONE, execution and gripper tests.
  • The CPython 3.12-only ARX ONE tests pass on 3.12 (64 passed, with the admission and tool tests).
  • The default tool specs are byte-identical to main at every commit.
  • scripts/validation/tool_composition.py: 8/8.
  • scripts/validation/release_contents.py accepts the built sdist and wheel.
  • The public content scan finds no credentials or machine-specific paths.
  • The remaining full-suite failures were checked one by one. None of them comes from these commits.

Cyberkona and others added 6 commits November 10, 2026 10:17
…s and opening checks

An environment adjusts the gripper and episode tools through its description
metadata; one that declares nothing keeps the previous schemas and behavior.

- finish_statuses: finish offers success, failure and give_up, recorded as
  completion_claimed, task_failed and task_abandoned. Default: success only.
- motion_profiles: replace whole named move profiles (max_step_m and
  arrival tolerances) with values matched to the arm's accuracy. The profile
  names stay fixed because they are part of the move schema.
- normalized_opening_tolerance: without a metric opening model, an explicit
  0-1 opening is checked against the measured 0-1 opening. Only readings the
  environment marks as measured are judged; others stay unknown.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--execution-mode returns with the values parallel and replanning (the former
serial value is not accepted). Unset, a run uses IntegrationSpec.execution_mode,
which is parallel unless an integration declares otherwise, so existing runs
keep their scheduling. Both modes accept every --tool-return-mode and
--resource-scheduling value.

replanning (execution/replanning.py) keeps parallel scheduling and adds:
- a world-mutating background call waits until the previous result on its
  resource has been read;
- when an action that may have moved something does not complete, queued
  world-mutating work on that resource is cancelled before dispatch, each
  with a receipt, and new work is refused (replan_required);
- work resumes once the failed result is read, the stop is confirmed and a
  replan_observation tool (capture_scene) returns a new snapshot. Unknown
  execution or an unconfirmed stop stays refused for the episode; finish
  (ends_episode) stays allowed. Nothing is retried or replayed.

Tools report what an action did in ToolOutput.action (ActionOutcome), which
is never sent to the model. Gripper tools fill it from motion feedback and the
final receipt's execution_outcome and stop evidence; a simulation counts as
stopped after a known result, an environment that declares physical_hardware
only when its receipt confirms the stop. ParallelToolExecutor gains small
extension points and is otherwise unchanged.

For all modes, cancel_tool and flush_queue return a receipt for each
cancelled job, and a refused call is recorded as a tool result.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An environment selects how the gripper tools move through its metadata
motion_execution:

- stepwise (default): the existing waypoint loop that checks tracking after
  every step. Environments that declare nothing are unchanged.
- wholepath: move sends one PathCommand that the environment preflights and
  runs without intermediate stops; arrival is checked once and a truncated
  path reports reachable_prefix. grasp, release and hold keep every joint at
  its measured angle in one command lasting steps control periods.

PathCommand is a typed protocol command (moving arms give a pose, held arms
give joints) served by an optional backend step_path; a backend without it
rejects the command. Tool schemas are the same in both modes; descriptions
follow the mode.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dual-arm ARX ONE (two ARX X5 arms, three RealSense D405 cameras) on the
shared Direct tools, and a template for other real robots:

- body/: the device process that owns the hardware. core/ and host/ are
  robot-independent; each device kind adds a driver and a host kit under
  drivers/ (fake/ is the smallest complete example).
- environment/: one physical episode over the Body RPC: admission, control
  lease, whole-path commands, observations and tracking recovery. The robot's
  shape (arms, joints, cameras) comes from environment.json.
- perception/: snapshots, RGB-D geometry, extrinsics and depth completion.
- offline/: the site fleet, read-only capture and hand-eye calibration, used
  outside episodes; deploy/: configuration templates and launch scripts.

The integration declares the replanning execution mode, finish statuses,
measured opening checks and its motion profiles, and uses the wholepath
motion mode when every Body arm serves a group path (environment.json can
force wholepath or stepwise). Physical results are never retried by the
episode. Values tuned for the X5 are marked robot-specific in the adapter.

Every episode confirms the arms are idle and takes the control lease before
it starts; there is no read-only episode mode. Read-only work uses the
offline capture and calibration tools.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the arx_one extra and package data for the integration's native sources
and templates, without local SDK copies or measured calibration. The release
contents check accepts the native driver sources, and the installed-SDK check
validates an arx_one run configuration alongside the other integrations.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ield

Recovery reached the model through two channels: the replanning executor's
replan report on results, and ARX ONE's own control_status state with a
different shape (capture_required, can_replan). The prompt described only the
second, so a model could stop for an operator after the executor had already
allowed the replan.

Environments now report facts and the executor alone decides:

- StepReceipt gains typed recovery facts: execution_outcome (none, partial,
  unknown), stop_confirmed and requires_operator_recovery. The primitive
  executor and the client backend carry them; the gripper tools turn them
  into ActionOutcome, which gains operator_required.
- replanning derives the only model-facing report from them. An operator
  block stops the plan even when nothing ran, and the unrecoverable phase is
  operator_required (was stop_unconfirmed). A failed action that ran nothing
  and needs no operator is an ordinary result and needs no new observation.
- A refused step that needs an operator reports stop_reason
  operator_recovery_required instead of planning_rejected.
- ARX ONE drops control_status and its replan state. A control lease that
  expires while the model thinks, or a confirmed tracking interruption, is
  taken again by the next command after a readiness check; an observation
  failure is only a rejected capture. Commands refused while movement is
  blocked return an operator receipt, and the prompt describes the replan
  field alone.

Environments that report none of the facts keep their previous behavior.

Co-authored-by: Yuan jiarui <118042747+1rubbishyuan@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Cyberkona

Cyberkona commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author
  1. refactor(replanning): report robot recovery only through the replan field. StepReceipt gains typed recovery facts, the replanning executor derives the only model-facing recovery report from them, and ARX ONE drops its own control_status recovery state.

Recovery reporting (commit 6)

Before this commit, recovery reached the model through two channels: the replanning executor's replan report on results, and ARX ONE's state.metadata.control_status with a different shape (capture_required, can_replan). The prompt described only the second, so a model could stop for an operator after the executor had already allowed the replan.

Environments now report facts and the executor alone decides:

Layer Change
services/environment/contracts.py StepReceipt gains execution_outcome (none/partial/unknown), stop_confirmed and requires_operator_recovery, all optional with defaults
services/environment/primitive.py, tools/gripper/motion.py Carry the facts through the service and client; a refused step that needs an operator reports stop_reason: operator_recovery_required instead of planning_rejected
tools/gripper/provider.py, tools/toolkit.py Turn the facts into ActionOutcome, which gains operator_required
execution/replanning.py One rule: an operator requirement, unknown execution, or partial execution without a confirmed stop → operator_required (was stop_unconfirmed); partial execution with a confirmed stop → observation_required → ready_to_replan; nothing ran and no operator needed → an ordinary failed result
integrations/arx_one/environment/adapter.py Drops control_status and its replan state. A lease that expires while the model thinks, or a confirmed tracking interruption, is taken again by the next command after a readiness check; an observation failure is only a rejected capture; commands refused while movement is blocked return an operator receipt
integrations/arx_one/prompt.py Describes the replan field alone

Effect on simulation

No behavior change for simulation under the default parallel mode:

  • Receipts: the simulation adapters that build step receipts (robodojo, robolab, robouse, libero_plus, calvin, robotwin; no other integration builds one) do not set the new fields. They keep the defaults: no execution report, no stop claim, no operator.
  • Stop handling: the tool layer still counts a stepped simulation as stopped after a known result, exactly as before.
  • Model-facing tools: the default tool specs are byte-identical to main. The aggregated feedback in tool results is unchanged.
  • Stop reasons: operator_recovery_required and the operator rule apply only when a receipt sets requires_operator_recovery. No simulation does.
  • Per-step replies: the service's per-step reply gains three keys; in simulation their values are null, null and false. For simulation, clients and servers from before this commit stay compatible in both directions: missing keys read as the defaults, and extra keys are ignored.

A simulation run with --execution-mode replanning keeps the same admission and cancellation rules. Two things in the report text differ:

  • After an unknown execution, the phase is named operator_required instead of stop_unconfirmed.
  • When a result has no error message, replan.reason is the feedback's stop_reason instead of action_incomplete.

Effect on ARX ONE

  • After an idle lease expiry, the model no longer has to capture before moving. The next command still rechecks idle state, device readiness and fresh feedback.
  • After a failed capture, motion stays available; the model can capture again or act on its last observation.
  • After partial execution, the model must still read the result and capture a new observation before the next action.
  • An operator block still refuses every command. The refusal is now a receipt (stop_reason: operator_recovery_required, replan.phase: operator_required) instead of an error message, and state no longer carries control_status.

@1rubbishyuan
1rubbishyuan merged commit 76c4cda into Open-RUA:main Nov 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants