Experimental MuJoCo session backend¶
robot_harness_mujoco.AlohaBackend connects the optional
Python session to the stepped
gym_aloha/AlohaTransferCube-v0 environment. The host owns the simulator, recording
and candidate queue. A separately supplied ACT worker receives RGB/joints and
returns candidates; model loading and task strategy stay outside this repository.
This is one qualified combination, not a generic MuJoCo robot adapter.
The current local qualification uses macOS arm64, Python3.12, MuJoCo3.8.1,
gym-aloha0.1.4 (revision bd3325740ea8d1c97411c41ea1e0f4ce0a7de8da),
gymnasium1.3.0, numpy2.2.6 and imageio2.37.4/imageio-ffmpeg0.6.0.
The external worker uses LeRobot revision
e595b7902714ba51f91e47523f66f89c5181b649, torch2.11.0 and the locally
migrated lerobot/act_aloha_sim_transfer_cube_human checkpoint at revision
ba73b2766f1371cdc133ca4efb97eb090d744625. These source identities describe
qualification, not a supported dependency matrix. The research consumer retains
its baseline; CMake does not install model dependencies.
The first Robot Agent delivery
now provides a bundled worker, Mac dependency lock and explicit weight preparation
in its pinned setup tutorial.
That installed Mac application passed its local normal path with new prepared
assets, reusing the qualified Harness installation. The first consumer increment
is merged as Agent06cbee4, with this runtime pinned at3acc9aa. It remains an
experimental combination, not a stable standalone skill interface or clean-machine
reproduction. Linux Ubuntu22.04/aarch64 CPU ACT and OSMesa passed the deterministic seed0
400+50-step path with a fresh environment/Harness installation,451 decoded
frames, two correlated settled receipts and process reaping. The same-run fixed
physical evaluator passed the bounded one-second hold. A separate installed Linux
visual application subsequently passed three real camera-based decisions, the
same400+50-step task, full recording, process exits and its own fixed physical
evaluation. Both results cover only the selected normal scope.
The portable session fixture needs none of these dependencies.
Consumer configuration¶
After building with ROBOT_HARNESS_BUILD_PYTHON=ON, use the matching interpreter
in an environment containing the qualified dependencies. The following values
are caller-supplied absolute paths, not files shipped by this package:
from robot_harness import Session
with Session(startup_timeout=90, mujoco={
"output": "/absolute/new-evidence-directory",
"worker_script": "/absolute/session_worker.py",
"checkpoint": "/absolute/migrated-checkpoint",
"device": "mps",
"seed": 0,
}) as session:
observation = session.observe() # RGB bytes plus camera/joints and sequence
submission = session.submit("trial-0", steps=400)
# Poll status("trial-0") until finished; retain its result and Core receipt.
# Then session.reset(seed=1) permits the next trial in the same host/worker.
Configuration is trusted local startup input, not a remotely supplied plugin
specification. The worker executable runs as the same local user. The output
directory must be new; existing evidence is never overwritten. Every episode
records the reset frame, action attempts before submission, post-step physical
facts, video and an execution summary. Ground-truth cube/contact facts stay in
these evaluator traces; the observation API exposes camera/joints only. After a
step error, observations are marked invalid and image sampling is refused.
steps counts confirmed returns; an errored call can already have had an effect,
which its best-effort error trace preserves.
The skill is aloha.transfer_cube_trial, at most400 native steps. Actions are14
absolute joint/gripper targets, validated in their native float32 representation.
No blanket normalized clipping is applied. An ACT chunk contains at most100
candidates; the host checks Core before each single native submission.
A native termination satisfying the environment criterion yields
native_condition_reached; exhausting the requested step budget yields
step_limit, native failure and no successful output. Full handoff and stable
holding remain unassessed: native reward4 does not prove right-hand release.
Cancellation freezes further stepped simulation once processed by the host;
physics is paused while awaiting inference. Settlement uses the explicitly scoped
mujoco-stepped-episode-v1 closure facts, not a physical stop or grasp guarantee.
Native-step stalls still stall the owner. Explicit reset is required after a
settled episode; a failed reset makes the session unavailable.
Validation¶
The local integration ran original seeds0–4 in one host/model process, matched all original actions and simulator traces exactly, retained all videos and confirmed process reaping. Actual simulation checks cover a3-step budget, nonempty-queue cancellation, and an error after an actual native effect. These are bounded integration evidence, not task reliability estimates.
Run the backend's lightweight numeric/cleanup tests without a model or renderer:
PYTHONPATH="$PWD/build-python/python" python3 -m unittest discover \
-s integrations/mujoco/tests -v
Only NumPy is needed for those tests. They and the standard session suite passed in a local Ubuntu22.04/Python3.10 container with distro NumPy1.21.5. The optional Ubuntu job includes the same checks; it does not run ACT or MuJoCo. Hosted workflow results are tracked in PR checks; those hosted fixtures do not establish Linux model/render results. The separate local Linux normal qualification above has its own scope. See Testing for current evidence boundaries.
Continuous handoff¶
Select mujoco-stepped-handoff-v1 in the trusted local mujoco configuration.
AlohaHandoffBackend then owns native dm_control continuation: it records reward4
as a milestone and executes the fixed400-step transfer segment without applying
Gym's legacy terminal mapping. After normal delivered settlement, it preserves
the episode for fresh RGB/joints and a separately admitted50-step hold.
The legacy mujoco-stepped-episode-v1 remains the default.
The Session interface requires explicit expected epoch/sequence references. Stage/resource/target checks govern hold admission; cube contact/pose truth remains evaluator-only recording data. A mistaken caller may hold a dropped cube target; the independent task evaluator must report that failure instead of Runtime quietly filtering it out.
Use the deterministic caller with the same qualified dependencies and externally supplied worker/checkpoint:
PYTHONPATH="$PWD/build-python/python" python3 examples/mujoco_handoff.py \
--output /absolute/new-handoff-run \
--worker-script /absolute/session_worker.py \
--checkpoint /absolute/migrated-checkpoint --device mps --seed 0
It performs both operations, checks continuous observations/settled receipts, rejects a stale hold reference, resets explicitly after hold, and closes/reaps the processes. It saves a plan and partial/error/final reports alongside episode recordings. Its status says whether the bounded execution checks completed; it does not report independently evaluated business success. It unconditionally selects hold as a deterministic baseline, not an observation-driven Agent.
A normal episode records450 returned steps and451 video frames. Per-operation summaries retain400/50 counts, epoch, skill and Core operation IDs; observation sequence and the raw trace count the whole episode. Transfer recording stays open through hold; close it before viewing the complete file. This profile is experimental and locally qualified only as described in Testing. The separate Robot Agent application adds observation-driven skill selection and its own preparation tutorial. Its visual assessment does not prove the physical task predicate. The deterministic Linux CPU/model/render path above passed; the separate Linux visual normal task also passed. Selected live failure cases and combination-wide delivery closeout remain subsequent work. No dependencies or model assets are installed by CMake.