TouchScale: 500 Hours of Human Vision and Touch for Visual–Tactile Learning

Dayou Li^1,*, Hao Wang^2,*, Qianqian Yang^3,*, Zihao Zhu^1,*, Haoquan Fang⁴, Ziyao Zeng⁵, Yan Han⁶, Zihan Wang⁷, Yan Wang⁸, Baoru Huang⁹, Dilin Wang¹⁰, Kenji Shimada³, Yiyue Luo¹¹, Manling Li¹², Teresa Lv¹³, Mustafa Mukadam¹¹, Rakesh Ranjan¹⁰, Ruohan Zhang⁴, Qi He⁶, Changliu Liu³, Xu Chen¹¹, Marco Pavone^4,8, Bangya Liu⁷, Jiachen Li¹⁴, Masayoshi Tomizuka¹⁵, Zhiwen Fan^1,†

¹Texas A&M University   ²Google DeepMind   ³CMU   ⁴Stanford University   ⁵Yale University   ⁶Microsoft   ⁷Overfit Lab   ⁸NVIDIA   ⁹University of Liverpool   ¹⁰Meta   ¹¹University of Washington   ¹²Northwestern University   ¹³Sony   ¹⁴Georgia Tech   ¹⁵UC Berkeley
^*Equal contribution   ^†Corresponding author

This repository is the official implementation of TouchScale. Mid-training on 500 hours of egocentric human vision and touch teaches robot policies to predict how touch will change.

Table of Contents

• Table of Contents
• Updates
• Overview
  • Scale and Task Diversity
  • Data Collection Setup
  • Mid-training Results
• Get Started
  • Repository Structure
  • Data Quality Control
  • N0-VTLA Mid-training and Post-training
• Acknowledgement
• License
• Citation

Updates

[2026-10-08] Released N0-VTLA with TouchScale mid-training and robot post-training. See n0-vtla/docs/MID_TRAIN.md and n0-vtla/docs/POST_TRAIN.md.

[2026-10-07] The TouchScale paper is available on arXiv.

[2026-10-04] Released the TouchScale data quality-control pipeline. See qc/README.md.

Overview

Scale and Task Diversity

500 hours across laboratories, kitchens, workbenches, and everyday environments.

Data Collection Setup

TouchScale is a large-scale egocentric visual-tactile dataset of everyday manipulation, recorded in laboratories, kitchens, workbenches, and other everyday environments. A single wearable setup records:

• a head-mounted RGB-D camera for the overall interaction,
• two wrist-mounted RGB cameras for close views of hand-object contact (both at 30 Hz),
• bimanual tactile gloves, each with 880 taxels across the five fingers and palm at under 2 mm spatial resolution, recording normal and shear force.

Mid-training Results

Mid-training on TouchScale teaches a policy to predict how touch will change before robot post-training. In real-robot experiments this raised task success from 22.5% to 57.5% on contact-rich tasks (soft/hard sorting, bottle-cap removal, test-tube transfer, whiteboard wiping). Scaling the data also improves zero-shot tactile contact prediction and egocentric action recognition.

Get Started

Repository Structure

Path	Description
qc/	The automatic data-quality pipeline used to screen recordings: a timestamp sync gate across all six sensor streams, and a VLM-assisted check for missing tactile signal and baseline noise. See qc/README.md.
n0-vtla/	The visual–tactile–action policy (a derivative of NeoteAI's N0-VTLA) with the TouchScale additions: action-free mid-training of the tactile branch on human TouchScale data, and post-training on your own robot recordings (data conversion, normalization scripts, launcher, real-robot inference notes). Start with n0-vtla/docs/MID_TRAIN.md and n0-vtla/docs/POST_TRAIN.md.

Data Quality Control

Full details are in qc/README.md.

1. Install dependencies (requires ffmpeg / ffprobe):

   cd qc
   pip install -r requirements.txt

2. Check timestamp sync across the sensor streams:

   python check_sync.py --root /path/to/recordings

3. Run the tactile QC (add a Gemini API key to .env first):

   cp .env.example .env
   python run_qc.py --data /path/to/recordings --out qc_out

N0-VTLA Mid-training and Post-training

Checkpoints are expected to live in a checkpoints/ folder next to n0-vtla/.

1. Install the environment: follow n0-vtla/docs/INSTALL.md.

2. Download the original N0-VTLA weights:

   cd n0-vtla
   hf download NeoteAI/n0-vtla-base --local-dir ../checkpoints/n0-vtla-base

3. Mid-train the tactile branch on TouchScale recordings (details: n0-vtla/docs/MID_TRAIN.md):

   python scripts/build_per_task_scale_normalization.py --raw-root /path/to/touchscale_raw --out per_task_scale.json --workers 16
   VTLA_ITW_RAW_ROOT=/path/to/touchscale_raw VTLA_ITW_NORMALIZATION=$PWD/per_task_scale.json \
     VTLA_PRETRAINED_CHECKPOINT=$PWD/../checkpoints/n0-vtla-base bash train_stage1.sh

4. Convert your robot recordings, then post-train (details: n0-vtla/docs/POST_TRAIN.md):

   # INIT=original   starts from the original checkpoint
   # INIT=touchscale starts from the original checkpoint + the mid-training result of step 3
   INIT=touchscale DATASET=/path/to/canonical_dataset ASSET_ID=my_asset bash scripts/posttrain.sh

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• N0-VTLA

License

Code in this repository is released under the MIT License, with one exception: the n0-vtla/ directory is derived from NeoteAI's N0-VTLA and remains under its original CC BY-SA 4.0 license (third-party notices in n0-vtla/NOTICE and n0-vtla/LICENSES/). The N0-VTLA model weights are governed by the Gemma Terms of Use.

Citation

If you find our work useful in your research, please consider giving a star ⭐ and citing the following paper 📝.

@misc{li2026touchscale500hourshuman,
      title={TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning},
      author={Dayou Li and Hao Wang and Qianqian Yang and Zihao Zhu and Haoquan Fang and Ziyao Zeng and Yan Han and Zihan Wang and Yan Wang and Baoru Huang and Dilin Wang and Kenji Shimada and Yiyue Luo and Manling Li and Teresa Lv and Mustafa Mukadam and Rakesh Ranjan and Ruohan Zhang and Qi He and Changliu Liu and Xu Chen and Marco Pavone and Bangya Liu and Jiachen Li and Masayoshi Tomizuka and Zhiwen Fan},
      year={2026},
      eprint={2610.10288},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.10288},
}