LoRA: Low-Rank Adaptation of Large Language Models

1 card

1. 01

   LoRA trains only a few small matrices beside the frozen model: that is why it costs one GPU and a few hours, not a cluster.

   LoRA: Low-Rank Adaptation of Large Language Models · 2021

   LoRA freezes the model and learns a small low-rank correction, so an adapter is a file, not a model.

   The mechanism: the change in the weights is approximated as the product of two low-rank matrices, with r far smaller than the layer's dimension. In the paper: 10,000 times fewer trainable parameters and 3 times less GPU memory than fully fine-tuning a 175-billion-parameter model. An adapter is a small file, swappable per task on the same base model; QLoRA adds a 4-bit quantised base. In 2026 a 7–8-billion model adapts in hours on a single GPU, from a few dollars for a small adapter to a few thousand for large datasets. What it changes: behaviour, format, style, tool-call discipline. What it does not: it adds no new knowledge in bulk — that is RAG. Serving many adapters on one base makes per-customer or per-task specialisation cheap.

   “We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.”

   Open the card →