SailNet classifies point of sail, tack and boat type from sailing video. It labels every frame of a sailing video with three attributes at once:
| Output | Classes | Notes |
|---|---|---|
| Boat type | ILCA, Optimist | the only two boats the model has seen |
| Side (tack) | port, starboard | |
| Direction (point of sail) | no-go zone, close-hauled, close reach, beam reach, broad reach, run | the head has six outputs, but only no-go zone, close-hauled and run were present in our footage |
It is a multi-task (multi-head) CNN [1] on an EfficientNetV2-B2 backbone [2], trained on 8,775 frames from 95 labelled clips. The per-frame labels are meant as the first layer of a manoeuvre-analysis system: tacks and gybes show up as transitions between labelled states.
Full write-up (dataset, architecture comparison, results, limitations): docs/Sailnet.pdf.
What this repository offers. The code of the pipeline (labelling and training notebooks) is provided so that the method can be inspected and re-run: with your own videos and labels you can retrain the models (see Reproducing the training). The videos the models were trained on are not released, so a run on your own data will not give the same numbers as the report.
Scope. The model was built for ILCA and Optimist footage only. Other boats were not considered in this work and are left for future work. The direction output is only meaningful for close-hauled and run. See Known limitations before using any prediction.
Held-out test set: 1,643 frames from clips never seen in training or validation. Final model: Model D after Phase 2 fine-tuning. Numbers are macro accuracy (mean of per-class recalls).
| Head | Macro accuracy |
|---|---|
| Side | 0.852 |
| Boat type | 0.884 |
| Direction | 0.594 |
Side (tack) is the most mature output and the one we would build on first. Boat type and direction are at an earlier stage: the footage covers two boat classes and three of the six points of sail, so broader and more varied data is the main next step. The current limits are listed in Known limitations and discussed in the report.
flowchart LR
A[Raw videos<br/>+ label per video] --> B[Frame extraction<br/>5 fps, 15 fps for no-go zone<br/>resize 224x224]
B --> C[Clip-level split<br/>61 / 19 / 15 clips]
C --> D[Phase 1: frozen backbone<br/>train neck + heads]
D --> E[Phase 2: unfreeze last<br/>10-15% of backbone]
E --> F[model_D_phase2.keras]
Labelling. Each training clip carries one label triple (point of sail, tack, boat type) applied to all its frames; the dataset has 95 clips. Frames are split by clip, not by frame, so near-identical neighbouring frames never end up in both train and test. The manifests carry no outing identifier, so different clips from the same outing can still land in different splits.
Architecture (Model D). Shared EfficientNetV2-B2 [2] (ImageNet weights) + a small neck, then three heads:
- boat type: reads the neck vector;
- side: reads the backbone feature map through a lightweight CBAM-style attention block [6] (port vs starboard depends on where the sail sits relative to the hull, which global pooling throws away);
- direction: reads the neck vector plus the boat-type logits.
Training uses a focal loss [3] with label smoothing [4] and a weighted sum of the three head losses (0.5 direction, 0.3 side, 0.2 boat). Frames are augmented on the fly with three fixed Albumentations [5] pipelines (no horizontal flip, which would swap port and starboard). We compared four ways of connecting the heads (A: independent, B: full cascade, C: partial cascade, D: partial cascade + side attention). The only change that clearly helped was the side head with attention on the unpooled feature map; the cascade connections gave no consistent benefit. See the report for the comparison.
.
├── README.md
├── LICENSE
├── requirements.txt
├── notebooks/
│ ├── Sailnet_Labeling.ipynb # videos -> frames -> train/val/test manifests
│ └── Sailnet_Training.ipynb # model A-D, Phase 1 / Phase 2, evaluation
├── docs/
│ ├── Sailnet.pdf # the report
│ ├── Sailnet_Presentation.pdf # slides
│ └── README.md # licensing note for the report and the slides
├── assets/ # figures used in this README
├── models/ # trained checkpoints: A-D after Phase 1, D after Phase 2
├── results/ # training logs of the reported run (see results/README.md)
├── CITATION.cff
└── .gitignore
The notebooks are meant to be read in this order: labeling first, then training.
models/ holds the five checkpoints of the reported run: Models A, B, C and D at the end of Phase 1, and Model D after Phase 2 (model_D_phase2.keras, the final model, about 72 MiB). Sizes and SHA-256 hashes of every file are listed in models/README.md.
The checkpoints are released under CC BY 4.0.
These come from the report, not from this README being cautious.
- Only ILCA and Optimist. Any other boat will be classified as one of the two. The boat-type head also generalised worse than validation suggested (test ILCA recall 0.77, with 23% of ILCA frames labelled Optimist).
- Direction knows three classes. Close reach, beam reach and broad reach have no training frames. Our expectation (not tested) is that footage sailed on those angles will be assigned to the nearest of the three known classes, so the direction column should not be trusted there.
- No-go zone is not recognised yet. This class has very little material: 13 training clips (about 22 seconds of real video) and only 3 test clips (69 frames). On the test set the model did not recognise it (recall 0.00), and with so few clips that estimate is itself very noisy. Scarcity of data is the most likely cause, but we have not verified it (that would take per-clip results). Until more footage is added, treat
no-go zonepredictions, and their absence, as unreliable. - Single frames, no temporal model. Predictions are independent per frame and are not smoothed. Detecting a tack or a gybe from the label sequence is future work.
- Small effective sample size. 95 clips in total, each model trained once with one seed, no confidence intervals. Differences of a few points between models should not be over-read.
- Possible optimism of the test scores. Clips from the same outing (same boat, place and light) can fall in different splits, so test clips may resemble training clips more than genuinely new footage would.
- Clip-level labels. In training, a clip containing a manoeuvre carries a single label for all its frames.
The notebooks are written for Google Colab with Google Drive, but also run locally. PROJECT_FOLDER, in the configuration cell of each notebook, is the only path to set. The names of the files inside it are aliases defined in the same cell (RAW_VIDEO_ZIP_NAME, JOBS_EXCEL_NAME, VIDEO_ZIP_NAME, DATA_ZIP_NAME, ...): rename them to match your own files. The raw videos, the video list, the extracted frames and the checkpoints are not included in this repository. To retrain on your own data, first install the dependencies with pip install -r requirements.txt (the notebooks ran on TensorFlow 2.20 / Keras 3.13):
- Put your videos in a zip (default name
raw video.zip) and list them in an Excel file (defaultvideo_jobs.xlsx, sheetjobs, columnsvideo,direction,side,boat_type,processed), both insidePROJECT_FOLDER. RunSailnet_Labeling.ipynb. It writes two archives intoPROJECT_FOLDER:video.zip(one folder of frames per clip) anddata.zip(train.csv,val.csv,test.csv,class_info.jsonand plots). - Run
Sailnet_Training.ipynbwith the samePROJECT_FOLDER. It findstrain.csvand the clip folders inside the two archives by itself, whatever their nesting. - The training notebook is shipped in results-only mode (
TRAIN_* = False,RELOAD_* = True): it expects the checkpoints and result files of the reported run, which are not in the repository. To train from scratch, setTRAIN_MODEL_X = TrueandRELOAD_MODEL_X = Falsefor each of the four models, thenTRAIN_PHASE2 = TrueandRELOAD_PHASE2 = False. The model sent to Phase 2 is set by hand in the selection cell (PHASE2_MODEL, default'Model D').
Requirements: ffmpeg for frame extraction (OpenCV is the fallback); Graphviz only for the architecture diagrams (skipped if missing). The reported run used Google Colab.
What can be reproduced is the method and the code, not the numbers of the report: the dataset of the reported run is not released, so a run on your own data will give different results. Even on the same data the numbers would not match bit for bit, because the augmentation parameters are random (the global seeds set in the notebook have no effect on Albumentations 2.x) and each model was trained once. The training logs of the reported run are described in results/README.md.
[1] S. Ruder, "An overview of multi-task learning in deep neural networks," arXiv preprint arXiv:1706.05098, 2017.
[2] M. Tan and Q. V. Le, "EfficientNetV2: Smaller models and faster training," in Proceedings of the International Conference on Machine Learning (ICML), 2021.
[3] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2999-3007, 2017.
[4] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the Inception architecture for computer vision," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[5] A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, and A. A. Kalinin, "Albumentations: Fast and flexible image augmentations," Information, vol. 11, no. 2, p. 125, 2020.
[6] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "CBAM: Convolutional block attention module," in Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Large language models were used as writing tools during this project, to help write and debug the code of the notebooks. The research questions, the experimental design, the choice of architectures and hyper-parameters, the interpretation of the results and the conclusions are the authors' own, and the authors checked the reported numbers against the experiments (same statement as in the report).
Nicolò Nucci and Chiara Mezzanzanica, Università degli Studi di Milano-Bicocca.
| What | License |
|---|---|
| Source code and notebooks | MIT |
Trained models (models/*.keras) |
CC BY 4.0 |
Report and slides text (docs/Sailnet.pdf, docs/Sailnet_Presentation.pdf) |
CC BY 4.0 |
Images: the figures in assets/ and all pictures and figures inside the report |
All rights reserved. No reuse without written permission |
Third-party material, if any, remains the property of its owners and is not covered by any of the licenses above. If you reuse the models or the report text, please cite the work as described in Citation.
If you use this work, please cite the report:
@misc{nucci2026sailnet,
title = {{SailNet}: Multi-Task Frame Classification},
author = {Nucci, Nicol{\`o} and Mezzanzanica, Chiara},
year = {2026},
note = {Universit{\`a} degli Studi di Milano-Bicocca}
}


