ARROW

Arbitrary Reconstruction and Tracking
of 4D Observations in the Wild

Ilya Fradlin    Christian Schmidt    Jens Piekenbrinck
Karim Knaebel    Gonzalo Martin Garcia    Bastian Leibe

RWTH Aachen University

Updates

Coming next: training code.

• 2026-11-03: Evaluation code is released.
• 2026-10-12: Inference code, pretrained weights, and an interactive viewer are released.
• 2026-10-01: Our paper is available on arXiv.

Overview

ARROW is a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. It combines the flexible point-query decoding of D4RT with support for arbitrary observations across viewpoints and capture times. This enables reconstruction and tracking from moving-camera videos, multiple video streams, and unordered photo collections.

ARROW achieves state-of-the-art 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, remaining competitive on 3D reconstruction tasks.

Visit the project page for interactive examples and results.

Installation

Clone the repository and install with uv:

git clone https://github.com/ifradlin/arrow.git
cd arrow
uv sync

This installs ARROW into .venv in editable mode. On Linux and Windows, uv sync installs PyTorch built for CUDA 13.0 (needs a recent NVIDIA driver). On macOS it uses Apple Silicon GPUs (MPS). ARROW picks CUDA, MPS, or CPU automatically. To choose a device, pass --device cpu (or cuda:1, mps) to arrow infer or arrow-viewer, or device= to ARROW.from_pretrained().

For the interactive viewer, add --extra viewer; for development (tests, linting, type checking), add --group dev.

Viewer

Run inference and view 3D reconstructions, point tracks, depth, and camera poses directly in the interactive viewer:

uv sync --extra viewer
uv run arrow-viewer

Open http://127.0.0.1:7860 to select input images or videos, choose tasks, and adjust run parameters such as frame selection and query spacing. Run inference and visualize the results directly in the viewer. See the viewer guide for input selection, run controls, layers, coloring, and filters.

Usage

Alternatively, track points from the CLI and use the viewer to visualize the saved predictions. This example takes every third video frame and uses the remaining defaults from configs/inference.py:

uv run arrow infer assets/videos/railway.mp4 --stride 3 --output workspace/results/railway.npz

The command saves workspace/results/railway.npz with the prepared input images, every query (pixel, source, target, and reference frame), its predictions (3D points, confidence, visibility, normals, ...), and metadata describing the frames and tasks. View the saved results in the viewer:

uv run arrow-viewer --scene workspace/results/railway.npz

This opens the saved predictions as a scene at http://127.0.0.1:7860. See all options for other tasks, grid spacing, and frame selection.

The same run from Python:

from arrow import ARROW
from arrow.infer import infer, load_input

model = ARROW.from_pretrained("ifradlin/arrow", device="auto")
observations = load_input("assets/videos/railway.mp4", stride=3)
result, encoded = infer(model, observations)

Weights download on first use. A local model directory also works; pass it to from_pretrained() or use --checkpoint /path/to/model. See inference examples for selected-point tracking, multi-view input, poses, and saved predictions.

Code structure

src/arrow/
├── models/              # Encoder and query decoder
├── infer/               # Input loading and inference
├── tasks/               # Reconstruction, tracking and pose queries
├── eval/
│   ├── preprocess/      # Dataset preparation
│   ├── dataset.py       # Shared evaluation data loading
│   ├── metrics/         # Numerical scoring and reductions
│   └── tasks/           # Evaluation protocols
└── viewer/              # Interactive viewer
configs/                 # Model, inference and evaluation settings
docs/                    # Usage and dataset guides
assets/                  # Example inputs and figures

Evaluation

1. Prepare the datasets. Follow the download and preprocessing instructions. The tracking evaluations use:

data/eval/
├── worldtrack/          # ADT, DynamicReplica, PointOdyssey, PStudio
├── tapvid3d/            # ADT, DriveTrack, PStudio minival
└── mvtracker/           # Kubric, DexYCB, PStudio

2. Run evaluation. Weights download automatically from Hugging Face.

# WorldTrack
uv run python -m arrow.eval --config configs/eval/benchmarks/worldtrack.py

# TAPVid-3D: RGB-only camera and world protocols
uv run python -m arrow.eval --config configs/eval/benchmarks/tapvid3d_camera.py
uv run python -m arrow.eval --config configs/eval/benchmarks/tapvid3d_world.py

# MVTracker: RGB-only multi-view tracking
uv run python -m arrow.eval --config configs/eval/benchmarks/mvtracker.py

Results are saved to workspace/results/<config-name>/summary.json and scenes.jsonl. See the evaluation guide for coverage, metrics, overrides and distributed runs. Additional pose, video-depth and reconstruction protocols use the same shared evaluation tasks.

Training

Coming next.

Citation

@article{fradlin2026arrow,
    title         = {{ARROW}: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild},
    author        = {Fradlin, Ilya and Schmidt, Christian and Piekenbrinck, Jens and Knaebel, Karim and Martin Garcia, Gonzalo and Leibe, Bastian},
    year          = 2026,
    journal       = {arXiv preprint arXiv:2610.01314},
}

License

See LICENSE for code and pretrained-weight terms. Weights are for non-commercial use and are also subject to the linked Waymo terms.