John Schulman et al.

1 card

1. 01

   The order is prompt, then RAG, then LoRA: fine-tuning pays when the behaviour is narrow and repeated, not when knowledge is missing.

   LoRA Without Regret · 2025

   Fine-tuning for behaviour, RAG for knowledge, a prompt for everything it can carry.

   The 2025 findings: LoRA on all matrices (including the MLP layers), a learning rate about ten times full fine-tuning's, two thirds of the compute, and for reinforcement learning rank 1 is enough. So "lower quality" is no longer the reason to avoid it. The decision tree: a better prompt with examples — free, instant; RAG — knowledge that changes, citations required; LoRA — a fixed format or tool-call discipline, style, a narrow classifier or router, latency (a short prompt instead of three thousand tokens of instructions), distilling a large model's behaviour into a small one for a single task (70–85% of the teacher's quality at 5–10 times lower cost, per 2026 reports). Not LoRA: new facts, changing policy, rare cases. Prerequisite: an evaluation set and hundreds or thousands of examples. The regret: the adapter freezes today's behaviour and is retrained when the model changes.

   “In our experiments, we find that indeed, when we get a few key details right, LoRA learns with the same sample efficiency as FullFT and achieves the same ultimate performance.”

   Open the card →