AutonomousGR

AutonomousGR is a ROS 2 Humble autonomy stack for an F1TENTH-style vehicle using a ZED 2i stereo camera. It detects track cones, estimates their 3D positions, builds a centerline, avoids depth obstacles, follows the safe path with pure pursuit, and supervises the final Ackermann command.

The stack targets:

• Jetson Orin Nano, CUDA, TensorRT, and ZED 2i on the vehicle
• NVIDIA Isaac Sim 6.0.1 on Ubuntu 22.04 with an RTX GPU
• ROS 2 Humble and ackermann_msgs/AckermannDriveStamped

System overview

Left image -> TensorRT cone detector -> depth localization -> left/right cones
                                                            |
                                                            v
                                                centerline /path/nominal
                                                            |
Registered depth -> inflated local obstacle grid -----------+
                                                            v
                                                  obstacle avoidance -> /path
                                                                        |
                                                                        v
                                                      pure pursuit -> /drive_candidate
                                                                        |
Obstacle grid + freshness + braking envelope ---------------------------+
                                                                        v
                                                         drive safety -> /drive

Core nodes:

• cone_detector_node: TensorRT/CUDA cone detection.
• cone_localizer_node: registered-depth projection into base_link.
• path_planner_node.py: Delaunay/greedy cone pairing and spline-smoothed nominal centerline.
• depth_obstacle_node: ground/hood filtering, point projection, grid inflation, and /obstacles/local_grid.
• obstacle_avoidance_node.py: laterally shifted paths constrained by obstacles and cone boundaries.
• pure_pursuit_node: steering and speed generation on /drive_candidate.
• drive_safety_node: sole /drive publisher; stops or limits speed using obstacle clearance and data freshness.
• stereo_depth_node: optional OpenCV SGBM depth generation for Isaac stereo qualification.

The externally important contracts remain:

/path   nav_msgs/Path in base_link
/drive  ackermann_msgs/AckermannDriveStamped

Repository layout

cone_nav/                  ROS 2 package, nodes, launch files, config, RViz, URDF
isaac_sim/                 Isaac scene, container launcher, and topic checks
Python Detection Code/     dataset conversion, training, and webcam utilities
fsoco.yaml                 YOLO dataset configuration
URCA Documents/            project documents

Requirements and model

Install normal ROS dependencies, then provide system-level CUDA, TensorRT, the ZED ROS 2 wrapper, and optional ORB-SLAM3:

source /opt/ros/humble/setup.bash
rosdep install --from-paths . --ignore-src -r -y

The trained YOLO26 checkpoint is included at models/fsoco_yolo26n/weights/best.pt. Export it on the target NVIDIA GPU:

python3 -m venv .venv-export --system-site-packages
source .venv-export/bin/activate
python -m pip install --upgrade pip ultralytics onnx onnxslim

yolo export \
  model=models/fsoco_yolo26n/weights/best.pt \
  format=engine imgsz=640 batch=1 dynamic=False \
  quantize=32 nms=False device=0 workspace=4

cp models/fsoco_yolo26n/weights/best.engine \
  cone_nav/models/cone_yolo.engine

The final engine belongs at:

cone_nav/models/cone_yolo.engine

The Isaac configuration expects the YOLO26 nms=False FP32 output [N,6] or [1,N,6] containing x1, y1, x2, y2, confidence, class. Legacy cx, cy, width, height engines remain selectable with trt_bbox_format:=cxcywh. Classes are 0=blue, 1=yellow, and 2=orange/big cone. TensorRT engines are GPU- and TensorRT-version-specific; build the production engine on the target platform.

Build

colcon build --packages-select cone_nav --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
source install/setup.bash

Run on the vehicle

ros2 launch cone_nav real.launch.py

This starts the ZED wrapper, cone pipeline, obstacle avoidance, safety supervisor, RViz, and ORB-SLAM3. Disable ORB-SLAM when it is not installed or calibrated:

ros2 launch cone_nav real.launch.py enable_orbslam:=false

The supplied ORB-SLAM YAML is only a starting point. Replace its intrinsics, stereo baseline, IMU noise values, and body-camera transform before trusting its output. The reactive planner itself does not require or consume ORB-SLAM odometry.

ZED wrapper topic names vary by release. Defaults use the repository's original names, while launch arguments support newer naming:

ros2 launch cone_nav real.launch.py \
  left_image_topic:=/zed2i/zed_node/left/color/rect/image \
  right_image_topic:=/zed2i/zed_node/right/color/rect/image \
  camera_info_topic:=/zed2i/zed_node/left/color/rect/camera_info

Confirm the installed wrapper's topics with ros2 topic list. Registered depth defaults to /zed2i/zed_node/depth/depth_registered.

Useful modes:

# Compute avoidance but publish the nominal path; drive safety remains active.
ros2 launch cone_nav real.launch.py avoidance_shadow_mode:=true

# Controlled regression bypass of avoidance and grid command checks.
ros2 launch cone_nav real.launch.py avoidance_enabled:=false

Run in Isaac Sim

The Isaac integration provides a Leatherback Ackermann vehicle, ZED 2i-like 0.12 m stereo pair, RGB, camera info, registered depth, /clock, cone corridor, collision obstacles, and native /drive control.

1. Start Isaac Sim

On an Ubuntu RTX host or Brev VM:

Allow inbound TCP 49100 and UDP 47998 from only the workstation that will run the WebRTC client. Do not expose the unauthenticated stream to the public Internet.

export ROS_DOMAIN_ID=0
export PUBLIC_IP=<host-public-ip>  # Only required for livestreaming.
ENABLE_LIVESTREAM=1 ./isaac_sim/run_brev.sh

For headless operation:

ENABLE_LIVESTREAM=0 ./isaac_sim/run_brev.sh --obstacle-scenario center

The launcher defaults to a 10 GB RTX 3080-safe camera profile: two 960x540 cameras at 20 Hz. Override it only after checking GPU memory usage:

ISAAC_RESOLUTION_WIDTH=1280 \
ISAAC_RESOLUTION_HEIGHT=720 \
ISAAC_CAMERA_RATE=30 \
ENABLE_LIVESTREAM=0 ./isaac_sim/run_brev.sh

Monitor the GPU in another terminal with watch -n 1 nvidia-smi. If Isaac Sim approaches the 10 GB VRAM limit, keep the default profile or reduce the camera rate to 15. The detector still uses a 640x640, batch-1 TensorRT input.

Available scenarios are clear, center, right, narrow, and blocked.

2. Start the autonomy stack disarmed

In another terminal on the same host:

source /opt/ros/humble/setup.bash
source install/setup.bash
ros2 launch cone_nav isaac.launch.py \
  enable_rviz:=false \
  engine_path:=/absolute/path/to/cone_yolo.engine

Isaac and all autonomy nodes must share ROS_DOMAIN_ID, Fast DDS, and simulation time.

3. Select the depth source

Ground-truth renderer depth is the deterministic default for functional and collision tests:

ros2 launch cone_nav isaac.launch.py depth_mode:=ground_truth

Stereo mode runs SGBM on the synchronized left/right images and publishes registered 32FC1 depth:

ros2 launch cone_nav isaac.launch.py depth_mode:=stereo

Use ground truth to validate navigation and stereo mode to expose calibration, texture, correspondence, and dropout problems. The vehicle uses the ZED SDK's stereo-derived registered depth.

4. Verify, then arm

./isaac_sim/check_topics.sh
ros2 topic hz /sim/camera/image_raw
ros2 topic hz /sim/camera/depth
ros2 topic hz /obstacles/local_grid
ros2 topic info -v /drive_candidate
ros2 topic info -v /drive

Only after RGB, depth, camera intrinsics, TF, detections, obstacle grid, empty-path stopping, and single /drive ownership are confirmed:

ros2 launch cone_nav isaac.launch.py \
  drive_enabled:=true \
  enable_rviz:=false \
  engine_path:=/absolute/path/to/cone_yolo.engine

Isaac defaults to a 0.5 m/s target and 1.0 m/s maximum. ORB-SLAM3 is disabled because the starter scene does not publish a calibrated /sim/imu.

Safety and tuning

The safety path is intentionally separate from planning:

• Invalid or out-of-range depth is discarded; ground and the vehicle hood are filtered.
• Obstacles are inflated before candidate-path collision checks.
• Candidate paths must remain inside the cone corridor.
• No feasible path produces an empty /path, which stops pure pursuit.
• Stale paths, grids, depth, or /drive_candidate messages produce zero speed.
• drive_safety_node is the only final /drive publisher.
• Speed is limited using v * latency + v^2 / (2 * deceleration) + margin.
• Isaac starts disarmed.

Primary tuning lives in params.yaml and isaac_params.yaml:

track_half_width: 0.75
lookahead_distance: 1.5
speed_target: 1.5
wheelbase: 0.32
obstacle_inflation_radius: 0.22
avoidance_max_lateral_offset: 0.55
safety_grid_timeout_sec: 0.20
safety_braking_deceleration: 2.0
safety_stop_margin: 0.30

Measure real braking performance before increasing vehicle speed. The defaults are starting values, not a validated physical safety case.

Legacy simulators

ros2 launch cone_nav sim.launch.py sim_type:=f1tenth
ros2 launch cone_nav sim.launch.py sim_type:=fsae

Legacy simulation bypasses obstacle avoidance by default because these bridges do not guarantee registered stereo depth. Enable it only after verifying /sim/camera/image_raw, right image, depth, camera info, timestamps, frames, and TF:

ros2 launch cone_nav sim.launch.py sim_type:=f1tenth avoidance_enabled:=true

The FSAE path uses ackermann_to_twist_node for a typed /drive to /cmd_vel conversion.

Validation

Source-level tests available on any machine:

python3 -m unittest discover -s cone_nav/test -p 'test_*.py' -v
bash -n isaac_sim/run_brev.sh isaac_sim/check_topics.sh

Bounded Isaac smoke test on the RTX host:

ENABLE_LIVESTREAM=0 ./isaac_sim/run_brev.sh \
  --test-steps 120 --obstacle-scenario center

Full compatibility requires a successful target colcon build, Isaac runtime test, ROS topic/type/rate checks, and disarmed stop-behavior validation. macOS cannot perform the ROS/Isaac runtime portion.

Training utilities

The repository retains lightweight dataset and model tools:

python3 "Python Detection Code/ConvertToYOLO.py"   # FSOCO -> three-class YOLO data
python3 "Python Detection Code/ConeDetection.py"  # train YOLO26
python3 "Python Detection Code/WebcamDetection.py" # quick camera/model check

Conversion reads ~/Downloads/fsoco_bounding_boxes_train by default; set FSOCO_DIR to use another source. It pairs .jpg, .jpeg, and .png images (case-insensitive extensions) with matching annotations, sorts them before the seeded train/validation shuffle, and clips boxes to the image before normalization and small-box filtering. Conversion checks that the source is a directory and discovers image/annotation pairs before replacing output. An invalid source, no matching pairs, or an error during discovery leaves existing output untouched. Once pairs are found, conversion deletes and recreates data/yolo_format, so preserve any needed output before rerunning it. This preflight does not validate every image or annotation; processing failures can still occur after replacement.

Annotation discovery tries <image-stem>.json, then <image-filename>.json, and accepts only files. Boxes with NaN or infinite coordinates are skipped; other valid boxes in the same annotation are still converted.

Converted image and label names share the original image stem plus a stable 16-character SHA-256 suffix derived from its path relative to FSOCO_DIR, including the source extension. This distinguishes repeated stems across team folders and image formats. Existing converted files are not renamed automatically.

Training initializes from models/fsoco_yolo26n/weights/last.pt when it is a file, otherwise best.pt, then yolo26n.pt if neither exists as a file. This loads weights into a new training run; the script does not request resume=True. It selects CUDA when available, otherwise CPU; set TRAIN_DEVICE to override that choice. Apple MPS remains disabled by default.

The webcam utility defaults to camera 0 and selects the first model file in its fallback list: YOLO26 best.pt, then legacy YOLOv8 best.pt and last.pt. Use --model to select a specific file and --camera for another camera index:

python3 "Python Detection Code/WebcamDetection.py" \
  --camera 1 --model /absolute/path/to/best.pt --conf 0.25 --imgsz 640

Press q in the preview window to quit.

Training outputs and datasets are intentionally ignored by Git. Only the TensorRT runtime contract and class mapping are required by cone_nav.

Known limitations

• The planner is reactive and camera-relative; it does not perform global-map planning.
• ORB-SLAM3 is optional and requires real stereo/IMU calibration.
• Isaac ground-truth depth validates navigation but not stereo correspondence quality; use depth_mode:=stereo for that.
• A TensorRT engine must match the target GPU, CUDA, and TensorRT versions.
• The stack uses standard ROS 2 messages and defines no custom interfaces.

The avoidance grid helpers reject nonfinite coordinates, origins, and resolutions, nonpositive resolutions, and coordinate conversions that overflow. Collision checks reject sampled path points outside the grid, in unknown or occupied cells, or in cells missing from the grid data. These checks apply at the sampled path points; they do not check the segments between points.