LocalText2SQL: QLoRA fine-tuning of Llama-3.2-3B for text-to-SQL

Fine-tunes Llama-3.2-3B-Instruct with QLoRA so it turns a question and a database schema into a SQLite query. The model is scored by running the SQL and comparing the results, not by comparing text.

Built with Llama. Adapter weights: https://huggingface.co/Sam-2610/Llama-3.2-3B-Text2SQL-QLoRA

Results

Evaluated on 150 questions the model never saw (50 easy, 50 medium, 50 hard):

	Untuned model	Fine-tuned	Change
Execution accuracy (all)	53% (79/150)	69% (104/150)	+16 points
Easy	78%	86%	+8
Medium	50%	78%	+28
Hard	30%	44%	+14
Valid SQL	87%	99%	+12

Of the questions where the two models differ, fine-tuning fixed 28 and broke 3. Reloading the published adapter in a fresh session reproduced the 69% result exactly. Per-question outputs are in results/.

How it works

1. Data (prepare_data.py): examples come from gretelai/synthetic_text_to_sql. Each one must run on SQLite and return real rows, and the data is balanced across three difficulty groups. Training uses 600 examples (200 per group) from the train split, and testing uses 150 (50 per group) from the separate test split. No question appears in both.
2. Training (train.py): 4-bit NF4 base model with LoRA adapters (r=16, alpha 32, dropout 0.05) on all attention and MLP projection layers, so 24.3M parameters (0.75%) are trained. 3 epochs, learning rate 2e-4 with a cosine schedule, effective batch size 16, FP16. Only the SQL answer contributes to the loss. Training took 15.5 minutes on a free Colab T4 GPU, and the loss fell from 0.51 to 0.08.
3. Evaluation (evaluate.py, utils.py): the schema and sample rows are loaded into an in-memory SQLite database, and both the correct and the predicted query are executed. They match if they return the same rows (order matters only when the correct query has ORDER BY). A query that fails to run counts as wrong.

Reproduce (needs an NVIDIA GPU; tested on a free Colab T4)

pip install -r requirements.txt
python prepare_data.py --out data
python evaluate.py --out results/baseline_results.json                         # untuned model
python train.py --train data/train.json --out adapter
python evaluate.py --adapter adapter --out results/tuned_results.json

Use the published adapter without training:

python demo.py --question "Who earns the most?"

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-3B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "Sam-2610/Llama-3.2-3B-Text2SQL-QLoRA")

Limitations

• The test questions come from the same synthetic dataset as the training data, so this shows the model learned this style of question, not that it is a general SQL expert. Public benchmarks such as Spider or BIRD would be a harder test.
• Hard questions (subqueries, window functions, multiple joins, set operations) are still only 44% correct.
• Each difficulty group has only 50 questions, so the per-group numbers are rough. The overall figure is more reliable.
• The dataset is synthetic and contains some noisy examples. Examples that cannot run on SQLite were removed.
• The model targets SQLite and short schemas. It does not handle very large databases.
• 4-bit training and inference require an NVIDIA GPU.

License and credits

• Code: MIT (see LICENSE).
• Model: built with Llama 3.2, subject to the Llama 3.2 Community License. Adapter weights are published under the same terms.
• Training data: gretelai/synthetic_text_to_sql by Gretel.ai, Apache 2.0.