SMARTFORM - A Vision-Based System for Exercise Form Guidance

An AI-powered fitness assistant that classifies exercises and evaluates exercise form in real time using Computer Vision, MediaPipe Pose, Deep Learning, and Autoencoders.

The system captures live video from a webcam, extracts human body landmarks, identifies the exercise being performed using a CNN+Transformer model, and evaluates exercise form using an Autoencoder-based anomaly detection pipeline trained exclusively on ideal exercise posture.

PROJECT GOAL

The objective of this project is to create an intelligent exercise guidance system capable of assisting fitness enthusiasts in maintaining proper exercise form while working out.

The system continuously monitors the user's movement through a webcam, identifies the exercise being performed, and evaluates whether the exercise is being executed with proper form. By providing real-time feedback, the system aims to reduce injury risk and improve exercise effectiveness.

THINGS THAT I WILL BE LEARNING FROM THIS PROJECT

• Human Pose Estimation using Computer Vision
• Real-Time Video Processing
• Deep Learning for Human Activity Recognition
• CNN-Based Feature Extraction
• Transformer-Based Sequence Modeling
• Autoencoder-Based Anomaly Detection
• Exercise Classification using Body Landmarks
• Form Quality Assessment using Reconstruction Error
• Data Augmentation for Pose-Based Datasets
• MediaPipe Pose Landmark Extraction
• Model Deployment and Real-Time Inference
• Building End-to-End AI Fitness Applications
• Integrating Multiple Deep Learning Models into a Single Pipeline
• Designing Real-Time Exercise Guidance Systems

FRAMEWORKS AND OTHER RESOURCES USED IN THIS PROJECT

• Computer Vision & Pose Estimation

  • OpenCV: Captures live webcam footage and processes image frames before pose estimation.
  • MediaPipe Pose: Extracts 33 body landmarks from each frame and provides x, y, z coordinates used by both the classification and posture assessment modules.

• Deep Learning

  • PyTorch: Used to build and train the CNN+Transformer exercise classification model.
  • TensorFlow: Used to build and train the Autoencoder responsible for form quality assessment.
  • Keras: High-level API used for designing and training the Autoencoder architecture.

• Machine Learning & Data Processing

  • NumPy: Performs numerical operations on extracted pose keypoints.
  • Pandas: Handles dataset management and preprocessing tasks.
  • Scikit-Learn: Provides StandardScaler for feature normalization and is used for preprocessing and evaluation utilities.
  • Joblib: Saves and loads preprocessing artifacts such as scalers and thresholds.

• Visualization & Experimentation

  • Matplotlib: Used to visualize training curves and evaluation metrics.
  • Jupyter Notebook: Used for experimentation, model development, and training workflows.

• Development Environment

  • Python: Primary programming language used throughout the project.

SYSTEM ARCHITECTURE

Exercise Classification Pipeline

Webcam
   ↓
OpenCV
   ↓
MediaPipe Pose
   ↓
33 Body Landmarks
   ↓
CNN + Transformer Model
   ↓
Exercise Classification

CNN+Transformer Model Pipeline

CNN+Transformer Model Model

Supported Exercise Classes:

• Deadlift
• Squat
• Lateral Raise

Form Quality Assessment Pipeline

MediaPipe Pose
   ↓
33 Landmarks (99 Features)
   ↓
Feature Scaling
   ↓
Dense Autoencoder
   ↓
Reconstruction Error
   ↓
Threshold Comparison
   ↓
Good Form / Bad Form

Autoencoder Model Pipeline

Autoencoder Model

DATASET

Exercise Classification Dataset

The classification model is trained using exercise videos collected from:

• MuscleWiki
• YouTube Exercise Demonstrations

The videos are processed into pose landmark sequences that are used to train the CNN+Transformer model.

Form Assessment Dataset

The Autoencoder is trained exclusively on examples of ideal exercise form.

Dataset Structure:

ideal-datasets/

├── Exercise1/
│   ├── frame1.jpg
│   ├── frame2.jpg
│   └── ...
│
├── Exercise2/
│   ├── frame1.jpg
│   ├── frame2.jpg
│   └── ...
│
└── ...

Each image is processed using MediaPipe Pose to generate:

33 Landmarks × 3 Coordinates
=
99 Features

The Autoencoder learns the distribution of correct exercise form and identifies deviations as anomalies.

The dataset s available at https://www.kaggle.com/datasets/hamiyasir/smartform-autoencoder-training-dataset

MODEL ARCHITECTURE

Exercise Classification Model

CNN + Transformer

The model combines:

• Convolutional Neural Networks (CNNs) for spatial feature extraction.
• Transformer layers for temporal sequence modeling.

Output Classes:

• Deadlift
• Squat
• Lateral Raise

Form Assessment Model

Dense Autoencoder

Input (99)
      ↓
Dense (64)
      ↓
Dense (32)
      ↓
Dense (16)
      ↓
Dense (32)
      ↓
Dense (64)
      ↓
Output (99)

The Autoencoder is trained only on ideal exercise posture.

Threshold Calculation:

threshold = mean(MSE) + 2 * std(MSE)

REPOSITORY STRUCTURE

Documentation

Contains:

• Research Paper
• Architecture Diagrams
• Model Evaluation Results
• Training Notes
• Project Reports

Exercise Classification

Contains:

• Dataset preprocessing scripts
• CNN+Transformer training code
• Model checkpoints
• Evaluation scripts

Form Assessment

Contains:

• Autoencoder training notebooks
• Feature extraction pipeline
• Saved scaler
• Saved threshold
• Autoencoder models

RESULTS

Exercise Classification

CNN+Transformer achieved:

Accuracy: 93%

for classifying:

• Deadlift
• Squat
• Lateral Raise

Form Quality Assessment

Dense Autoencoder achieved:

Accuracy: 86%

for identifying:

• Correct Form
• Incorrect Form

FUTURE IMPROVEMENTS

• Add support for additional exercises.
• Introduce exercise-specific posture feedback.
• Deploy on mobile devices.
• Implement voice-based coaching.
• Add repetition counting.
• Support multi-person exercise tracking.
• Integrate workout analytics dashboards.
• Expand dataset diversity for improved generalization.

DEPLOYMENT

This project has yet to be deployed :/