What "fine-tuning" costs by default

Fine-tuning a pretrained language model the direct way updates every one of its weights:

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
total_params = sum(p.numel() for p in model.parameters())
print(f"{total_params:,}")

76,961,152

Nearly 77 million parameters, and a full fine-tune's optimizer has to track gradients and momentum state for every single one of them, real memory overhead on top of the model itself, whether or not most of those weights actually need to change much for the specific task at hand.

What LoRA actually changes

LoRA (Low-Rank Adaptation) freezes the entire original model and inserts small, trainable matrices alongside a subset of the existing layers, training only those:

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    r=8, lora_alpha=16, lora_dropout=0.05,
    target_modules=["q", "v"],
)
lora_model = get_peft_model(model, lora_config)
lora_model.print_trainable_parameters()

trainable params: 344,064 || all params: 77,305,216 || trainable%: 0.4451

344,064 real trainable parameters, out of 77.3 million. Everything else in the model stays frozen, exactly as pretrained, contributing to the forward pass but never receiving a gradient update. The optimizer only needs to track state for that 0.45%, not the whole model.

Whether that shows up as real, measured speed

Parameter count is one claim. Whether it translates into faster training is a separate, checkable one. Timing five real training steps on the same task and the same hardware, full fine-tune against LoRA:

full_times = time_training(full_model, steps=5)
lora_times = time_training(lora_model, steps=5)

full fine-tune: 0.69s, 0.45s, 0.40s, 0.38s, 0.39s -> average 0.46s/step
LoRA:           0.26s, 0.18s, 0.18s, 0.18s, 0.18s -> average 0.19s/step

A real, measured 2.4x speedup per training step, on CPU, for this model and this batch size. The gap would widen further on GPU-based training with larger models, where the optimizer memory savings (not just per-step compute) become the dominant real cost difference, but even here, with a small model on a CPU, the difference is directly measurable, not theoretical.

Why freezing most of the model doesn't wreck quality

The part that makes this work rather than just being a shortcut with a quality cost: a pretrained model already encodes most of what a downstream task needs, general language understanding, grammar, broad world knowledge. Most fine-tuning tasks are a comparatively narrow adjustment on top of that, not a rebuild from scratch. LoRA's bet, validated repeatedly in real practice since the technique was published, is that a small number of additional trainable parameters, inserted at the right layers, is enough to express that narrow adjustment, without needing to touch the much larger body of knowledge the frozen weights already represent.

The takeaway

"LoRA trains a small fraction of the parameters" is usually stated as a fact to take on faith. Here, checked directly against a real model: 0.45% of the total, confirmed via print_trainable_parameters(), and a real, measured 2.4x per-step speedup that follows directly from it. The claim holds up against direct measurement, which is exactly the bar worth applying before trusting any performance claim about a technique before adopting it.