Climbing the maze — Q-learning and DQN in Gymnasium

Course project for Planning and Decision Making (PADM), M.Sc. AI Engineering of Autonomous Systems, Technische Hochschule Ingolstadt (summer semester 2026).

Two reinforcement-learning agents that learn to navigate a maze:

	Task 1 — task_1_custom_env/	Task 2 — task_2_dql/
Environment	Custom discrete grid world (12 × 5), built from scratch	Continuous 2D maze (course-provided layout)
Algorithm	Tabular Q-learning	Deep Q-Network (PyTorch)
State	Cell (x, y)	Position (x, y) ∈ [0, 1]²
Actions	8: move or climb × 4 directions	4: up, down, left, right

Task 1 — custom grid world + tabular Q-learning

Environment

A Gymnasium environment (engine/MyEnv.py) with pygame rendering:

• 29 walls between cells, 6 danger cells (episode ends), 3 bonus cells and a goal.
• Besides moving, the agent can climb: a climb ignores walls but costs more. The agent has to learn when a shortcut over a wall is worth it.
• Bonuses can be collected once per episode.

Event	Reward
Move	−0.01
Climb	−0.1
Bonus (once per episode)	+0.5
Danger cell	−20 (terminal)
Goal	+10 (terminal)

Training

• Q-learning with γ = 0.99, random start positions (random_initialization = True).
• Exploration: two schedules compared, ε = max(0.999ⁿ, 0.1) and an oscillating ε = sin²(n) / ln(n + 2).
• Learning rate: compared 3e-4, 1e-2 and 1e-1. 1e-2 was used for the final runs; with 3e-4 the Q-table hadn't converged after 10,000 episodes.
• Early stopping: the Q-table is snapshotted every episode into two alternating 100-episode windows. Training stops when the average tables of the two windows differ by less than a threshold. In practice the threshold was hard to tune, so the final runs used a fixed 10,000 episodes.

Result

Learned greedy policy and max Q-value per cell (α = 0.01, decaying ε, 10,000 episodes). Arrows are moves; double arrows are climbs. Brown lines are walls, red frames are bonus cells.

A bug found along the way. Early runs produced Q-values above 100 near the goal, which is impossible when the goal is worth +10. The cause was the update for terminal transitions: it still bootstrapped from max Q(next_state). Because the agent could also start on the goal cell, Q(goal) fed back into itself and grew towards 10 / (1 − γ). Using target = reward for terminal steps fixed it (commit aa56ad7); values are now bounded by the goal reward, as in the plot above.

Task 2 — continuous maze + Deep Q-Network

The environment (env.py) was provided by the course: walls and danger zones are fixed, and the goal is a circle on the right. I designed the reward function and the network, and tuned the training.

Event	Reward
Each step	−distance to goal (dense shaping)
Bumping into a wall / border	−1 (agent stays in place)
Danger zone	−10 (terminal)
Goal	+10 (terminal)

• Network: fully connected 2 → 128 → 128 → 64 → 4, ReLU. An earlier 2 → 64 → 64 → 4 net also solved the task.
• Training: experience replay (50k buffer, batch 128, 10 gradient steps per episode), target network synced every 50 episodes, Adam (lr 3e-4), γ = 0.98, ε decays 0.9995 per episode down to 0.02 once the buffer holds 2,000 transitions, max 400 steps per episode, Huber loss.
• The DQN training loop is adapted from minimalRL.

Result

The trained agent (experiment 6) steering around the central danger zone to the goal:

With a fixed start (experiment 6) the agent learns to reach the goal reliably within a few thousand episodes. Random starts were harder. The 64 → 64 net (experiment 3, γ = 0.99) met the stopping criterion (mean reward of the last 10 episodes > 9) after about 9,800 episodes. Experiment 5 (larger net, γ = 0.98) stayed noisy for all 20,000 episodes. Part of that is the environment: starts are sampled anywhere in the square, including inside walls, where most moves count as collisions. An agent stuck there collects −1 per step, which is the flat band at −400 (400 steps × −1) in the right-hand plot.

Run it

pip install numpy gymnasium pygame matplotlib seaborn torch

# Task 1: train + visualise (switches at the top of main.py)
cd task_1_custom_env && python main.py

# Task 2: set train_dqn / test_dqn / render and exp_no at the top of main.py
cd task_2_dql && python main.py   # default: watch the trained agent (dqn_6.pth)

Task 1 writes each run to experiments/experiment_N/ (Q-table, parameters, plots).