Local Runtime MCP integration¶
Use an existing tool-using agent, such as Codex, to inspect and operate one Core-backed Python Session. The server provides local stdio tools; it owns one Session client, while the existing host and policy worker continue progressing separately. This is experimental source packaging, not a published plugin, remote service or stable cross-version API.
Two Agent paths¶
| Consumer | Owns | Runtime connection |
|---|---|---|
| Existing general-purpose agent, such as Codex | Its model/context and decisions using the available tools | This optional MCP module |
| Embodied business Agent we implement for a concrete scenario | Business goal, task state, interpretation of continuous observations, skill selection, memory and bounded recovery | Direct Python/native interfaces, or MCP when useful |
| ACT policy worker | Observation-to-joint-action prediction | Existing host-owned policy lane; neither a business Agent nor a tool-using task Agent |
The first path validates tool consumption. It does not deliver the second path, whose task logic and application packaging belong outside Robot Harness. Both consume the same execution/observation/receipt semantics. Native control and device protection remain below Runtime; neither Agent can allocate Core authority.
First run without a model or simulator¶
Requires POSIX, CMake3.18+, a C++17 compiler, Python3.10+ with development headers, and the optional SDK/image dependencies. Core builds remain independent of them. From this repository:
python3 -m venv build-mcp-venv
build-mcp-venv/bin/python -m pip install -r integrations/mcp/requirements.txt
cmake -S . -B build-mcp -DROBOT_HARNESS_BUILD_PYTHON=ON \
-DBUILD_TESTING=ON -DPython3_EXECUTABLE="$PWD/build-mcp-venv/bin/python"
cmake --build build-mcp --target _core --parallel 2
PYTHONPATH="$PWD/build-mcp/python" build-mcp-venv/bin/python \
examples/mcp_session.py --output "$PWD/build-mcp/example-01"
Expect status: completed, two operations with values7 and4, accepted output and
settled receipts, the same host/worker across explicit reset, and confirmed close.
The example uses the real SDK/stdin/stdout/Core path. It is a deterministic caller,
not a live model result. It generates report.json, server/tools.jsonl and stderr
logs in the new output directory. Use a different output path on the next run;
existing evidence is never overwritten. Fixture observations have no robot image.
The package is copied beside robot_harness in the build and optional installation.
After cmake --install, point PYTHONPATH at the installed
<prefix>/lib/robot-harness/python (or the configured CMAKE_INSTALL_LIBDIR). The
matching interpreter must have the SDK packages installed; copying source does
not install dependencies or provide a cross-version binary ABI.
Configure the ALOHA simulation¶
Prepare the qualified backend dependencies and supply your ACT worker and checkpoint. The product does not download weights or install LeRobot. Save an operator-owned JSON file, with absolute paths:
{
"backend": "mujoco",
"worker_script": "/absolute/skill/session_worker.py",
"checkpoint": "/absolute/models/migrated",
"device": "mps",
"seed": 0,
"startup_timeout": 90
}
Run the same SDK example using the matching skill environment/interpreter:
PYTHONPATH="/absolute/build-mcp/python" /absolute/skill-env/bin/python \
/absolute/robot-harness/examples/mcp_session.py \
--config /absolute/aloha.json --output /absolute/new-mcp-example
It performs two400-step-budget trials, resetting to seed1 between them. Results
and videos are under server/episodes/; four real camera samples become PNGs in
server/. Native-condition success is limited to the declared upstream criterion;
full right-hand release and stable holding remain unassessed. device selects the
external worker device, not proof that a different GPU/OS combination is qualified.
The current model/rendering qualification remains macOS arm64/MPS.
Without --config, the module selects the counter fixture. An explicit fixture
configuration may set worker_delay_ms and startup_timeout. Unknown optional
metadata is ignored; required backend names, path types and consumed values are
validated. There is no arbitrary backend import or remote executable tool.
Connect Codex or another stdio client¶
Any MCP client can launch the following server using the matching interpreter and environment. Do not run it as an interactive application expecting terminal text; stdout is reserved for protocol traffic, diagnostics go to stderr.
PYTHONPATH="/absolute/build-mcp/python" /absolute/skill-env/bin/python \
-m robot_harness_mcp --config /absolute/aloha.json \
--output /absolute/new-server-run --max-trials 2 --max-tool-calls 80
For Codex, adapt this local configuration example to your paths. It follows the
official MCP configuration.
The fixture needs no --config argument or simulator environment.
[mcp_servers.robot]
command = "/absolute/skill-env/bin/python"
args = ["-m", "robot_harness_mcp", "--config", "/absolute/aloha.json", "--output", "/absolute/new-server-run", "--max-trials", "2"]
startup_timeout_sec = 20
tool_timeout_sec = 120
required = true
[mcp_servers.robot.env]
PYTHONPATH = "/absolute/build-mcp/python"
Each server process needs a fresh output path, including a newly launched
connection. A connection cannot reopen after close_session; start a new server
when needed. This module does not modify global client configuration, authentication
or credentials. The operator configures the consumer model and authorizes task
effects. Tool annotations are client hints, not enforcement or a sandbox.
A first task can ask the agent to inspect capabilities and an actual observation, perform one supported trial, follow status, then report the native outcome, output and settlement separately. Ask it to explain whether full handoff is supported. Explicitly authorize a second trial/reset if desired. Task prompts and Agent strategy belong to the consumer; the server itself contains no planning loop.
Tools and cancellation¶
| Tool | Behavior |
|---|---|
open_session |
Open the fixed operator configuration; repeated calls retain the same Session |
capabilities |
Read actual skill/availability, native criterion and unsupported guarantees |
observe |
Latest measurements; camera backends return PNG plus epoch/sequence/age |
submit |
Supported skill, caller request ID,1–400 steps and1–60000 ms deadline |
status |
Session progress or a retained operation by ID |
cancel |
Request task revocation; follow status for native completion/settlement |
reset |
Explicit reset after settlement; optional ALOHA seed0–4 |
close_session |
Close admission and confirm owned process reaping; retry an uncertain close |
MCP request cancellation and robot-task cancellation are different. Cancelling a
queued MCP call prevents it from starting. Cancelling a call already using the
synchronous Session waits for that call to finish safely; it does not undo a
submitted operation. Query its ID and explicitly call cancel if required. Native
revocation starts when the host processes it, not when the model sends a request.
One already-entered synchronous native step and host stalls have no hard stop bound.
EOF closes the owned Session. Startup failure cannot silently spawn a replacement; failed cleanup cannot claim successful close. The default bounds are three distinct supported submission IDs and80 tool calls, adjustable by the operator to1–64 and 1–10000 respectively. Refused/uncertain submissions consume an ID slot; duplicate IDs still reach Runtime's existing deduplication. Close remains available after budget exhaustion. No reconnect recovery or persistent task state is provided.
Verification¶
PYTHONPATH="$PWD/build-mcp/python" build-mcp-venv/bin/python \
-m unittest discover -s integrations/mcp/tests -v
The tests include real stdio/Core tasks, reset/old receipts, strict pre-effect argument rejection, queued task cancellation and EOF during active work. A controlled startup barrier also exercises actual MCP cancellation notifications: the started open completes without a duplicate protocol response, a cancelled queued submit has no effect, subsequent status works and processes are reaped. Focused checks cover failed-startup non-replacement, failed-close retry, budgets and invalid images. They require no model or renderer. The Ubuntu optional-Python job includes these checks and an installed/relocated MCP example. See Testing for executed evidence and limits.