What is the difference between Low-Rank Adaptation (LoRA) and full fine-tuning of large language models?

Question

Machine Learning — Hard

What is the difference between Low-Rank Adaptation (LoRA) and full fine-tuning of large language models?

Accepted Answer

Low-Rank Adaptation (LoRA) reduces trainable parameters by injecting low-rank decomposition matrices into attention layers, while full fine-tuning updates all model parameters. This allows LoRA to achieve comparable performance with reduced memory requirements.