[rylinjames]

Why

On one L4, Tether's pi05 decomposed path disagreed with LeRobot eager (FP32 CUDA: mean cosine 0.77, min -0.33), and the default FP16 TensorRT path crashed on the first predict with MemcpyFromHost 'Memcpy_token_961': Unsupported type proto value case:0.

Fixes

1. Prompt state. LeRobot's pi0.5 processor builds Task: <task>, State: <256-bin ints>;\nAction: from the normalized state. Tether always dropped the state. The exporter now records decomposed.state_in_prompt from the policy: false for state-out students (state_proj) and for use_proprioceptive_memory configs. The server reads that field. For older exports it falls back to not expert_takes_state. An export that puts state in the prompt now refuses a request that has no state. The misleading comment is fixed.
2. Two more input divergences found during the rerun. Both are verified against LeRobot 0.6.1 locally.
   • PaliGemma's tokenizer pads left by default; LeRobot pads right. The server now sets right padding.
   • The prefix graph feeds images straight into embed_prefix, so SigLIP needs the [-1, 1] range. Tether sent [0, 1] images after a PIL antialiased resize. _prep_image is now a NumPy port of resize_with_pad_torch followed by *2-1. Max deviation from torch is 2.4e-7.
   • Tokens match LeRobot exactly on 11 of 11 states, including out-of-range bins.
3. Action dim. The export wrote action_dim=32 (the padded size). It now writes output_features["action"] (7) and also records decomposed.max_action_dim. For older exports the server takes the action dim from the normalizer stats. It refuses a config whose action dim disagrees with the stats.
4. TensorRT crash, root cause. With optimize=False, the expert export leaves 370 SplitToSequence+SequenceAt pairs. Neither the TensorRT parser nor the CUDA EP takes SplitToSequence, so ORT 1.25 runs it on the CPU and runs SequenceAt on CUDA. TensorRT's MemcpyTransformer treats CUDA nodes as its own device. It inserts a TensorRT MemcpyFromHost for every sequence; the INFO log shows Add MemcpyFromHost after split_N for TensorrtExecutionProvider 370 times. That kernel copies only tensors.
   • _repair_exported_onnx now rewrites each pair to a single Split. Outputs are bit-identical on ORT CPU.
   • The provider plan refuses TensorRT, with the reason, for graphs that still contain sequences.
5. TensorRT precision. Once it ran, FP16 TensorRT gave wrong actions (cosine 0.079); BF16 gave 0.988. Exports now record source_dtype, and bfloat16-trained checkpoints keep TensorRT at FP32/TF32.
6. The teacher normalizer fetch is pinned to model_revision.

Evidence (one L4, 24 paired observations and noise vs LeRobot eager)

arm	p50 / p95 / p99 ms	max abs	mean abs	cosine mean / min
Tether CUDA FP32	460.8 / 468.6 / 472.1	1.6e-2	1.1e-3	0.999992 / 0.999973
Tether TRT FP32/TF32 (new default)	441.8 / 454.5 / 456.6	1.6e-2	1.1e-3	0.999992 / 0.999973
Tether TRT FP16 (old default, after the crash fix)	384.9 / 386.7 / 388.4	2.84	0.325	0.079 / -0.219
LeRobot eager (bf16 weights)	535.2 / 548.4 / 552.8	-	-	-

Full write-up: tether-studio-context 03_experiments/2026-10-27-pi05-l4-instinctflash-vs-tether/RESULTS.md, section "Rerun after fixes".

Not done here

• The prefix still runs on CUDA EP: TensorRT is disabled for >8 GB external-data graphs. ORT 1.25's trt_load_user_initializer may remove that limit. It is untested here.
• The residual of about 9e-4 mean abs against an FP32 LeRobot reference does not come from TF32 (measured). Its cause is still unknown.

Tests

uv run --no-sync pytest tests/ -q -p no:cacheprovider: 4702 passed, 121 skipped (main baseline: 4679 passed). Each new test fails on main.

🤖 Generated with Claude Code