“In our experiments, we find that indeed, when we get a few key details right, LoRA learns with the same sample efficiency as FullFT and achieves the same ultimate performance.”John Schulman et al. · LoRA Without Regret · 2025 · LoRA Without Regret (Thinking Machines Lab, 29 septembrie 2025) — introducere
Fine-tuning for behaviour, RAG for knowledge, a prompt for everything it can carry.
The 2025 findings: LoRA on all matrices (including the MLP layers), a learning rate about ten times full fine-tuning's, two thirds of the compute, and for reinforcement learning rank 1 is enough. So "lower quality" is no longer the reason to avoid it. The decision tree: a better prompt with examples — free, instant; RAG — knowledge that changes, citations required; LoRA — a fixed format or tool-call discipline, style, a narrow classifier or router, latency (a short prompt instead of three thousand tokens of instructions), distilling a large model's behaviour into a small one for a single task (70–85% of the teacher's quality at 5–10 times lower cost, per 2026 reports). Not LoRA: new facts, changing policy, rare cases. Prerequisite: an evaluation set and hundreds or thousands of examples. The regret: the adapter freezes today's behaviour and is retrained when the model changes.
The card says the right order is prompt, then RAG, then LoRA. But the less-traveled part is what comes before all three: a way to measure whether the new behavior is actually what you want. Without that, you cannot tell if a better prompt would have sufficed. A concrete example: you want the model to classify support tickets into five categories. First, write twenty tickets with the correct answer. Only then try a prompt with examples. If it misses half, move to RAG or training. The evaluation set is the compass, not a luxury. And LoRA in 2025 no longer loses quality versus full fine-tuning, so the old brake is gone. What remains is the practical question: what exactly must be configured for it to work well in use?
Passage 1 ended with a question: what do you configure so LoRA works well? The 2025 answer is counterintuitive. Don't apply it only to attention layers; apply it to all the model's matrices, including the MLP ones. And raise the learning rate roughly ten times above the usual full fine-tuning value. Why: the adapter is a small part of the model, so it needs bigger steps to learn anything. Concrete example: you train the ticket classifier; you use the full fine-tuning learning rate and get a model that seems to have learned nothing. You raise the rate tenfold and the behavior takes hold. Even rank 1 suffices for reinforcement learning, at two thirds of the compute. So the question is no longer whether LoRA is good enough, but when training is worth it at all.
Why it matters Without an evaluation set, a fine-tune is an expense with a story attached.