How LoRA Actually Works: Low-Rank Decomposition, Weight Merging, and Memory Breakdown Under the HoodIf you try to full-p...

How LoRA Actually Works: Low-Rank Decomposition, Weight Merging, and Memory Breakdown Under the HoodIf you try to full-parameter fine-tune an 8-billion parameter language model in 16-bit precision, your GPU memory requirement immediately explodes past 80 GB.The raw model weights only occupy 16 GB of VRAM ($8 \times 10^9 \text{ parameters} \times 2 \text{ bytes}$). Yet standard training crashes on a single 80 GB A100 or H100.Where does the missing 64+ GB of VRAM go? And why can Low-Rank Adaptation (LoRA) reduce that memory footprint by over 70% while updating fewer than 0.1% of the parameters and retaining 99% of full fine-tuning performance?Here is what actually happens mathematically, mechanically, and in physical GPU memory during LoRA fine-tuning and inference. 1. The Real Memory Culprit: Why Full Fine-Tuning ExplodesWhen training a neural network with the standard AdamW optimizer, model weights are only a tiny fraction of your memory consumption.For every single trainable parameter...