LoRA: Low-Rank Adaptation for Fine-Tuning Large Models

Motivation & Problem Setting

Full fine-tuning of modern foundation models updates all $|\Theta |$ parameters, where $|\Theta |$ can reach $10^{11}$. Each downstream task produces a full-sized checkpoint, making per-task storage, distribution, and serving infeasible. We want a method that (i) reduces trainable parameters by orders of magnitude, (ii) does not add inference latency, and (iii) matches full fine-tuning quality. LoRA — introduced by Hu et al. (2021, Microsoft) — is currently the dominant answer.

Why not just freeze + linear probe?

Linear probing is too weak for generative tasks (no internal representations adapt). We need a method that adapts internal computations of every transformer block, but cheaply.

Why not adapters?

Adapters (Houlsby et al., 2019; Pfeiffer et al.) insert small bottleneck MLPs $\sigma (x{W}_{\text{down}})W_{\text{up}}$ inside each transformer block. They work, but the inserted modules are sequentially on the forward path → inference latency overhead, especially painful at batch size 1 in autoregressive decoding. LoRA's key engineering win is that its reparameterization can be algebraically merged into the base weight at inference time.

Core Mathematical Formulation

The LoRA Reparameterization

Let $W_0\in {\mathbb{R}}^{d\times k}$ be a frozen pre-trained weight matrix (e.g., $W_Q$, $W_V$ in attention). Standard fine-tuning learns

$$W=W_0+\Delta W,\Delta W\in {\mathbb{R}}^{d\times k}.$$

LoRA constrains $\Delta W$ to be low-rank:

$$\Delta W=BA,B\in {\mathbb{R}}^{d\times r},A\in {\mathbb{R}}^{r\times k},r\ll \min (d,k).$$

The forward pass becomes:

$$h=W_{0}x+\Delta Wx=W_{0}x+B(Ax).$$

Parameter Count

• Full FT of one matrix: $dk$ params.
• LoRA: $r(d+k)$ params.
• For $d=k=4096$, $r=8$: $\frac{r(d+k)}{dk}=\frac{8\cdot 8192}{16,777,216}\approx 0.0004$ — a ~2500× reduction for that matrix.

Initialization Scheme

$$A\sim \mathcal{N}(0,{\sigma }^2),B=0.$$

Hence at step 0: $\Delta W=BA=0$, so the model behaves exactly like the frozen base. Training starts at the pre-trained solution and moves outward.

Scaling Factor $\alpha /r$

LoRA introduces a scalar $\alpha$ and uses:

$$\Delta W=\frac{\alpha }{r}BA.$$

The original paper sets $\alpha$ once and varies $r$, claiming this approximately removes the need to retune the learning rate when sweeping $r$. The intuition: as $r$ grows, $\Vert BA\Vert$ tends to grow proportionally; dividing by $r$ keeps the effective update magnitude scale-invariant.

Theoretical Foundations

The Intrinsic-Rank Hypothesis

LoRA's central conjecture: the update matrix $\Delta W$ acquired during fine-tuning has low intrinsic rank, even though it sits in an ambient space of dimension $dk$.

Empirical evidence in the paper:

• Random projection to $r=1$ or $r=2$ subspaces sometimes nearly matches $r=64$.
• The top singular directions of $\Delta W$ amplify features that are already present but underused in $W_0$ (i.e., $\Delta W$ correlates with task-relevant singular directions of $W_0$ that have small singular values).

This builds directly on Aghajanyan et al. 2020 "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" which showed that one can re-parameterize the entire fine-tuning trajectory in a $\sim 200$-dimensional subspace and recover ≥90% of full-FT performance for BERT-base. Larger pre-trained models have smaller intrinsic dimension — the opposite of what naive parameter-counting would suggest.

Expressivity of Rank-$r$ Updates

Any matrix in ${\mathbb{R}}^{d\times k}$ of rank $\le r$ can be written as $BA$ with the given shapes. So LoRA's hypothesis class is exactly the rank-$\le r$ matrices. This is a strict subset of ${\mathbb{R}}^{d\times k}$. When $r=\min (d,k)$, LoRA recovers full fine-tuning capacity (but loses parameter savings).

The implicit prior: $\Delta W$ should be a low-rank perturbation. This is a strong inductive bias and is also a source of LoRA's limitations (see Biderman et al. critique below).

Gradient Dynamics

For loss $\mathcal{L}$:

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial B}& =\frac{\partial \mathcal{L}}{\partial \Delta W}{A}^{\top }\\ \frac{\partial \mathcal{L}}{\partial A}& =B^{\top }\frac{\partial \mathcal{L}}{\partial \Delta W}\end{array}$$

Note the asymmetry: $A$'s gradient is left-multiplied by $B^{\top }$, which is small early in training (init at 0). $B$'s gradient depends on $A^{\top }$, which is full-rank random Gaussian. This creates a feature-extraction/feature-projection split: $A$ identifies useful input directions, $B$ projects them into output space. The asymmetry motivates LoRA+ (Hayou et al. 2024), which argues $B$ should have a much larger LR than $A$ (often ${\lambda }_B=16{\lambda }_A$) to be in the feature-learning regime rather than the lazy regime.

Which Weights to Adapt?

Standard Application

The original paper applies LoRA only to attention weight matrices, and within attention, finds:

Targets	Quality
$W_Q$ only	Weak
$W_V$ only	Decent
$W_Q+W_V$	Strong (best param-efficient)
All of $\{W_Q,W_K,W_V,W_O\}$	Marginally better, doubled params
FFN layers	Often substantial gains in modern practice

In modern instruction tuning (LLaMA, Mistral), it is now standard to apply LoRA to all linear projections (attention + MLP up/gate/down). This is what HuggingFace PEFT defaults to via target_modules="all-linear".

Embeddings and Output Heads

LoRA is usually not applied to token embeddings or LM head. If your task requires new tokens (e.g., role markers <|user|>), you typically train those embeddings with full precision separately — they're cheap.

Implementation Sketch (PyTorch)

import torch
import torch.nn as nn
import math

class LoRALinear(nn.Module):
    def __init__(self, base_linear: nn.Linear, r: int, alpha: float, dropout=0.0):
        super().__init__()
        self.base = base_linear            # frozen
        for p in self.base.parameters():
            p.requires_grad = False
        in_f, out_f = base_linear.in_features, base_linear.out_features
        self.A = nn.Parameter(torch.empty(r, in_f))
        self.B = nn.Parameter(torch.zeros(out_f, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        # base path (frozen)
        out = self.base(x)
        # LoRA path
        out = out + (self.dropout(x) @ self.A.T @ self.B.T) * self.scale
        return out

Two design choices worth noting:

• Dropout on the LoRA branch only (regularizes the low-rank update).
• The base forward and LoRA forward are independent computational graphs — only LoRA params accumulate gradients.

Major Variants

QLoRA

Dettmers et al. 2023. Combines:

1. 4-bit NormalFloat (NF4) quantization of the frozen base $W_0$.
2. Double quantization of the quantization constants themselves.
3. Paged optimizers to handle memory spikes from gradient checkpointing.
4. Standard LoRA on top of the quantized base.

Forward pass: dequantize $W_0$ block-wise on-the-fly, compute $W_{0}x$, add $BAx$. Backward pass: gradients flow only into $B,A$ (the quantized base is non-differentiable).

NF4 is theoretically the information-optimal 4-bit datatype for normally-distributed weights (which pre-trained LLM weights approximately are, post-LayerNorm), based on quantile quantization.

DoRA — Weight-Decomposed LoRA

Liu et al. 2024. Decomposes $W$ into magnitude and direction:

$W=m\cdot \frac{V}{|V{|}_c},m\in {\mathbb{R}}^{1\times k},,V\in {\mathbb{R}}^{d\times k}$

where $|\cdot {|}_c$ is column-wise L2 norm. Then $V$ is LoRA-decomposed ($V_0+BA$) and the magnitude $m$ is learned directly. Empirically narrows the gap to full FT, especially at low rank. Interpretation: full FT adjusts both magnitude and direction; vanilla LoRA only adjusts a low-rank combination of both, which is sub-optimal when the task primarily requires magnitude recalibration.

AdaLoRA

Zhang et al. 2023. Instead of fixing rank $r$ uniformly, parameterize $\Delta W=P\Lambda Q$ (SVD form) and prune $\Lambda$'s singular values during training via an importance score. Allocates more rank to layers/heads that need it. The bookkeeping is non-trivial and gains are modest; less popular in production than DoRA.

LoRA+

Hayou et al. 2024. Pure theory result: in the infinite-width limit, $A$ and $B$ require different learning rates to be in the feature-learning regime. Empirically, ${\lambda }_B/{\lambda }_A\in [4,16]$ improves convergence speed and final quality, especially for harder tasks. Drop-in compatible with any LoRA codebase.

VeRA — Vector-based Random Matrix Adaptation

Kopiczko et al. 2024. Freezes $A,B$ as random projections shared across layers and only learns per-layer scaling vectors $b,d$:

$\Delta W_{\ell }=\text{diag}(b_{\ell })B,\text{diag}(d_{\ell })A$

with $A,B$ random and identical across layers. Reduces trainable params by another order of magnitude vs. LoRA, with surprisingly small quality loss. Echoes the Lottery Ticket Hypothesis family — most of the work can be done by selecting/scaling random features.

LoHa, LoKr

LyCORIS family (originally from diffusion fine-tuning). Replace the $BA$ outer product with Hadamard (element-wise) or Kronecker products of two low-rank factorizations. Higher effective rank at the same param count. More common in image generation (Stable Diffusion LoRAs) than in LLM land.

Comparison Table — PEFT Methods

Method	Trainable params	Inference latency	Mergeable	Modifies architecture
Full FT	$	\Theta	$	0
Adapter (Houlsby)	~3%	↑	No	Yes (inserts modules)
Prefix tuning	~0.1%	↑ (longer seq)	No	No (input-side)
Prompt tuning	~0.01%	↑ slight	No	No
BitFit	~0.05% (biases only)	0	Trivially	No
LoRA	~0.1%–1%	0	Yes	No
QLoRA	~0.1%–1%	0	Yes (after dequant)	No (base is quantized)
DoRA	~LoRA + small	0	Yes	No
VeRA	~0.01%	0	Yes	No

Limitations & Recent Critiques

LoRA Learns Less and Forgets Less

Biderman et al. 2024. Controlled study on code and math fine-tuning. Headline findings:

• LoRA underperforms full FT on out-of-distribution generalization for hard domains (e.g., code generation, math).
• LoRA preserves the base model's behavior on tasks outside the adaptation domain better than full FT — i.e., less catastrophic forgetting.
• Both effects come from the same source: the low-rank constraint limits how far the model can move from $W_0$.

Effective Rank ≠ Nominal Rank

Empirical analyses show the effective rank of LoRA-learned $\Delta W$ (top-$k$ singular values containing 95% of energy) is often much smaller than nominal $r$. So setting $r=64$ vs $r=8$ may not give you a meaningfully more expressive update — the optimizer simply doesn't use the additional capacity. Implication: simply cranking $r$ up isn't a free lunch.

Catastrophic Interference in Multi-Task LoRA

Composing multiple LoRAs (e.g., averaging two task adapters' $\Delta W$) often produces a model that is worse at both tasks than either individually. The space of low-rank updates is not closed under addition in a task-preserving sense. See LoRA Hub (Huang et al.), Mixture-of-LoRAs, and the broader Model Merging literature (Task Arithmetic, TIES Merging, DARE).

Connection to Adjacent Topics

vs. Sparse Fine-Tuning

Sparse FT (e.g., LT-SFT, FISH Mask): learn $\Delta W$ as a sparse matrix rather than a low-rank one. Same goal (parameter-efficient), different inductive bias. Sparse FT is harder to implement efficiently on GPU because sparse matmul kernels are slower than dense ones, despite the smaller flop count. This is why low-rank (which uses dense small matmuls) dominates in practice — hardware bias, not algorithmic superiority.

vs. RLHF Fine-Tuning

RLHF (or DPO) on top of LoRA is now standard: SFT-LoRA → DPO-LoRA on a frozen base. Memory savings compound, but be aware that DPO's reference model is the SFT-LoRA-merged model, which adds bookkeeping. The reward model itself is usually a separate model (often LoRA-tuned too).

This is directly relevant to safety fine-tuning: many RLHF and Constitutional AI pipelines train safety-relevant heads or behaviors via LoRA, which means the safety modification is a low-rank perturbation and thus potentially fragile — see the literature on LoRA jailbreaks and the Shadow Alignment paper (Yang et al. 2023) showing that 100 examples of bad LoRA training can undo safety alignment. For your line of work on safe-by-construction AI: the fact that alignment lives in a low-rank subspace that can be easily perturbed by another low-rank update is a structural alignment fragility worth taking seriously.

vs. Mechanistic Interpretability

Recent work (e.g., Sharkey et al., Bushnaq et al. on circuits) is starting to ask: what subspace does LoRA modify? If you can characterize the low-rank update's singular directions in terms of pre-existing model features (SAE directions, attention head functions), you can predict generalization. This connects to Linear Representation Hypothesis and the broader project of treating fine-tuning as a tractable, low-dimensional intervention rather than an opaque parameter shift.

Practical Recipe (Default Settings, May 2026)

Final Synthesis

LoRA's enduring impact comes from a clean factorization of two concerns: (1) the inductive-bias claim that fine-tuning lives in a low-rank subspace, which is theoretically grounded in intrinsic-dimension work, and (2) the engineering claim that low-rank + mergeable at inference are the two properties that matter for production. Variants (QLoRA, DoRA, VeRA, LoRA+) refine each axis without abandoning the framework. The most interesting open questions are not "can we make $r$ smaller" but "what does the low-rank update actually represent, and what behaviors are categorically out of reach for any low-rank intervention?" — and the latter is where the alignment/safety community is now circling.

Memory Cheatsheet: Training vs Inference (with LoRA)

Quick mental model: in inference you only pay for weights + activations + KV cache. In training you also pay for gradients, optimizer states, and activations have to be kept around for backward. That last bit is the killer.

The base unit: bytes per parameter

Precision	Bytes/param
fp32	4
fp16 / bf16	2
int8	1
int4 / NF4	0.5

So a 7B model in bf16 weights = $7\times 10^9\times 2=14$ GB. Memorize this — everything else is multipliers on top.

Inference memory

Roughly three buckets:

Total ≈ Weights + KV Cache + Activations(small)

Weights: params × bytes_per_param. 7B bf16 → 14 GB. 70B bf16 → 140 GB.

KV cache (the sneaky one for long contexts): $\text{KV}=2\times L\times n_{\text{kv-heads}}\times d_{\text{head}}\times \text{seq\_len}\times \text{batch}\times \text{bytes}$ The 2 is for K and V. For LLaMA-2-7B (L=32, 32 heads, d_head=128, bf16) at seq_len=4096, batch=1: $2\times 32\times 32\times 128\times 4096\times 1\times 2\approx 2.1$ GB. Doubles every time you double context.

Quick trick: for a 7B-class model, KV ≈ 0.5 MB per token per batch in bf16. So 8k context, batch 4 = ~16 GB just in KV. This is why GQA / MQA exist.

Activations: tiny at inference (you discard them layer by layer). Ignore unless you're being pedantic.

Training memory — the 4× rule

Rule of thumb for full fine-tuning with AdamW in mixed precision:

Total ≈ Weights + Gradients + Optimizer + Activations
      ≈ 2P     + 2P        + 8P        + activations
      ≈ ~16 bytes/param  +  activations

Where the 8P for Adam = two moments (m, v) in fp32 = 4+4 bytes, plus often a fp32 master copy of weights = another 4. People bookkeep this slightly differently, but ~16–20 bytes/param is the right ballpark.

So a 7B model needs ~112 GB just for the static stuff — already over an A100-80GB. Add activations and you're toast. This is why nobody full-FTs 7B on one GPU without tricks.

Activations during training scale as: $\text{act}\sim \text{batch}\times \text{seq\_len}\times L\times d_{\text{model}}\times \text{(some multiplier 10-30)}$ The multiplier depends on what gets saved for backward (attention scores hurt the most — quadratic in seq_len without FlashAttention). With gradient checkpointing, you trade ~30% compute for activations going from $O(L)$ down to $O(\sqrt{L})$ — typically cuts activation memory by 5–10×.

The LoRA cheat

LoRA changes the picture dramatically because only the adapter params get gradients + optimizer states. Let $P$ = base params, $P_{\text{LoRA}}$ = LoRA params (typically 0.1–1% of $P$).

LoRA training ≈ 2P (frozen weights)        ← bf16, no grads
              + 16 × P_LoRA                ← grads + Adam on tiny matrices
              + activations                ← still depend on full forward!

Concrete: 7B in bf16, LoRA with r=16 on all linears, ~40M trainable params:

• Frozen weights: 14 GB
• LoRA grads + optimizer: $16\times 40\text{M}\approx 0.6$ GB
• Activations: maybe 5–15 GB depending on batch/seq

Total ~20–30 GB. Fits an A6000 or even a 3090 with checkpointing.

QLoRA — the further cheat

Quantize the frozen base to 4-bit NF4:

• Frozen weights: $7\text{B}\times 0.5=3.5$ GB (was 14)
• LoRA grads + optimizer: ~0.6 GB
• Activations: same

Total ~10–15 GB. This is why a 65B model fits on one A100-80GB — it goes from ~130 GB frozen to ~32.5 GB frozen.

Quick estimation workflow

When someone asks "will this fit?":

1. Weights: params × bytes (2 for bf16, 0.5 for 4-bit).
2. Mode:
   • Inference → add KV cache (≈ 0.5 MB/token for 7B-class, scales with model size).
   • Full FT → multiply weights cost by ~8 (so bf16 weights × 8 ≈ training static cost).
   • LoRA → just weights cost + ~1 GB for adapters.
3. Activations: rough guess batch × seq × 0.5 MB for 7B-class without checkpointing; divide by ~5 with checkpointing.
4. Add 10–20% slop for fragmentation, CUDA workspace, etc.

Sanity check examples

Setup	Memory
7B inference, bf16, seq=2k, bs=1	~16 GB
7B inference, bf16, seq=32k, bs=1	~30 GB (KV dominates)
7B full FT, bf16, bs=4, seq=2k	~110 GB + activations → multi-GPU
7B LoRA, bf16, bs=4, seq=2k	~25 GB → single 3090/A6000
7B QLoRA, bs=4, seq=2k	~12 GB → single 4090
70B QLoRA, bs=1, seq=2k	~45 GB → single A100-80GB

The mental shortcut I actually use: bf16 inference ≈ 2× params in GB. Full FT ≈ 16× params in GB. LoRA training ≈ inference cost + a bit. QLoRA ≈ inference cost / 4.

Practical Indications

When LoRA is Feasible

Low-rank updates tend to be sufficient when:

• the pretrained model already contains the necessary capabilities;
• fine-tuning primarily needs to select, redirect, or reweight those capabilities;
• the new task is close to pretraining;
• the dataset is relatively small.

They are less reliable when adaptation requires:

• learning a large amount of new domain knowledge;
• continued pretraining on billions of tokens;
• major changes to representations;
• difficult mathematics, code or reasoning specialization;
• adapting a weak base model to a substantially different target distribution.

Comparing to RAG

Requirement	Better default
Teach response style or behaviour	LoRA
Teach domain terminology and recurring patterns	LoRA
Store a moderate, stable body of knowledge	LoRA may work
Store thousands of changing facts	RAG
Require citations or source traceability	RAG
Correct or delete one fact quickly	RAG or model editing
Work without retrieval infrastructure	LoRA
Combine internalised behaviour with current facts	LoRA + RAG

LoRA as Parametric Knowledge Memory

What It Means for Information to Be “Inside” a LoRA

Knowledge Is Stored in the Adapter–Backbone Interaction

For an adapted model,

$$f_{{\Theta }_0,\varphi }(x),\varphi =\{A_{\ell },B_{\ell }{\}}_{\ell \in \mathcal{S}},$$

the base parameters ${\Theta }_0$ remain frozen, while $\varphi$ denotes the LoRA parameters attached to the selected modules $\mathcal{S}$.

The learned knowledge is therefore distributed across:

• Base representations: entities, syntax, semantic features, reasoning primitives.
• Adapter parameters: task-specific redirection, association, suppression, or recombination.
• Prompt: the key used to activate the relevant modified computation.
• Decoder dynamics: the autoregressive process that reconstructs the answer.

Consequences:

• The same adapter attached to a different base-model revision may fail.
• Removing the adapter usually removes the newly acquired behaviour.
• Merging $BA$ into $W_0$ does not change the represented function; it only changes deployment form.
• A small adapter can exploit a large amount of pre-existing structure in the base model.
• Adapter size alone does not measure the total complexity of the resulting model.

A better conceptual expression is:

$$\text{learned response}=\text{base knowledge}+\text{adapter intervention}+\text{prompt-conditioned retrieval}.$$

Residual Information Rather Than Total Information

The adapter need not encode an entire fact from scratch. It must encode only the information that is not already predictable from the frozen model.

Let $Z$ denote the target fact and ${\Theta }_0$ the frozen model. The relevant information burden is closer to:

$$H(Z\mid {\Theta }_0),$$

rather than the unconditional entropy $H(Z)$.

Examples:

New information	Residual burden on the adapter
“Paris is a city”	Almost none; already strongly represented
“Project Helios uses protocol XJ-17”	Entity–attribute association may be genuinely new
A random 10-digit number assigned to a fictional name	High-entropy arbitrary association
A new response style	Low-dimensional behavioural change
A new mathematical technique	Potentially requires knowledge, representations, and procedures
Reversing a well-established fact	Must overcome an opposing prior in the base model

This is analogous to Conditional Kolmogorov Complexity: the relevant question is not “How long is the fact?” but “How difficult is the fact to specify given what the model already knows?”

Memorization, Internalization, and Capability Acquisition

These terms should not be treated as synonyms.

Process	Operational meaning	Typical evaluation
Verbatim memorization	Reproduce an exact target sequence	Exact match
Associative memory	Recover a value from a learned key	Key–value recall
Factual internalization	Answer semantically equivalent questions about a fact	Paraphrase QA
Knowledge revision	Replace or qualify an existing model belief	Counterfactual editing
Relational generalization	Apply learned relations to unseen combinations	Held-out compositions
Capability acquisition	Learn a reusable procedure or algorithm	Out-of-distribution task performance
Behavioural adaptation	Modify style, policy, format, or preference	Behavioural benchmark

A LoRA may achieve high training-set exact match while failing factual internalization. Conversely, it may learn a useful behaviour without storing many discrete facts.

Parameter Count, Rank, and Functional Capacity

Counting LoRA Parameters Correctly

Raw Trainable Parameters

For one matrix $W\in {\mathbb{R}}^{m\times n}$,

$$\Delta W=BA,B\in {\mathbb{R}}^{m\times r},A\in {\mathbb{R}}^{r\times n}.$$

The raw trainable parameter count is:

$$P_{\mathrm{raw}}=r(m+n).$$

Across a set of adapted modules $\mathcal{S}$:

$$P_{\mathrm{LoRA}}=\sum _{\ell \in \mathcal{S}}{r}_{\ell }\left(d_{\mathrm{in},\ell }+d_{\mathrm{out},\ell }\right).$$

Under fixed module placement and uniform rank:

$$P_{\mathrm{LoRA}}\propto r.$$

Doubling $r$ therefore doubles the number of stored scalar parameters, but it does not imply doubling useful memory.

Independent Degrees of Freedom

The factorization $BA$ is non-identifiable because, for any invertible matrix $G\in {\mathbb{R}}^{r\times r}$,

$$BA=(BG)(G^{-1}A).$$

Multiple $(A,B)$ pairs therefore represent the same update matrix.

The manifold of rank-$r$ matrices has dimension:

$$P_{\mathrm{DoF}}=r(m+n-r).$$

The difference is:

$$P_{\mathrm{raw}}-P_{\mathrm{DoF}}=r^2.$$

Interpretation:

• $r(m+n)$ counts stored trainable scalars.
• $r(m+n-r)$ counts the effective local degrees of freedom of the resulting matrix.
• $r^2$ degrees correspond to redundant changes of basis within the latent rank-$r$ space.
• At small $r$, this distinction is minor.
• At large $r$, raw parameter count increasingly overstates functional freedom.

Scaling $\alpha$ Does Not Add Capacity

For:

$$\Delta W=\frac{\alpha }{r}BA,$$

$\alpha$ changes the scale of the accessible update but not the rank of $\Delta W$.

Therefore:

• Increasing $\alpha$ can improve optimization when updates are too weak.
• Excessive $\alpha$ can destabilize training or overwrite base behaviour.
• Increasing $\alpha$ does not create new independent directions.
• A rank-$8$ adapter with large $\alpha$ remains rank at most $8$.

This distinguishes:

• Capacity limitation: insufficient update directions.
• Optimization limitation: available directions exist but are not learned effectively.
• Scaling limitation: update directions are learned but have insufficient or excessive magnitude.

Worked All-Linear Parameter Estimate

Consider a hypothetical 32-layer transformer with:

• model width $d=4096$;
• MLP width $d_{\mathrm{ff}}=14336$;
• query and output widths $4096$;
• grouped-query key/value widths $1024$;
• LoRA applied to $Q,K,V,O$, gate, up, and down projections.

Parameters per layer are:

$$\begin{array}{rl}{P}_{\mathrm{attn}}& =r[(4096+4096)+(4096+1024)\\ & +(4096+1024)+(4096+4096)]\\ & =26424r,\end{array}$$

and:

$$P_{\mathrm{MLP}}=3r(4096+14336)=55296r.$$

Hence approximately:

$$P_{\mathrm{total}}\approx 32(81720r).$$

At $r=16$:

$$P_{\mathrm{total}}\approx 41.8\text{ million parameters}.$$

The thing to remember is that this number says how many scalars are optimized—not how many independent facts will be recalled reliably.

Empirical Capacity Laws

What Research Currently Supports

Rank-Dependent Capacity and Saturation

Back et al.'s Understanding LoRA as Knowledge Memory studies LoRA ranks from $2$ to $1024$ and controlled knowledge loads from approximately $1,000$ to $20,000$ tokens using arbitrary phone-book associations and counterfactual facts. [S1]

The principal findings are:

1. Absolute memory capacity increases with rank.
2. Every fixed rank exhibits a finite saturation point.
3. The saturation point moves upward as rank increases.
4. Performance can collapse sharply after capacity is exceeded.
5. Marginal gains diminish at high ranks.
6. Maximum absolute capacity and maximum parameter efficiency occur at different ranks.

Define a thresholded capacity:

$$K_{\tau }(r)=\max \left\{K:\mathrm{Score}(r,K)\ge \tau \right\},$$

where:

• $K$ is the knowledge load;
• $\tau$ is an acceptable recall threshold;
• $K_{\tau }(r)$ is the largest load supported before saturation.

Parameter efficiency can then be defined as:

$$E_{\tau }(r)=\frac{K_{\tau }(r)}{P_{\mathrm{LoRA}}(r)}.$$

Empirically:

$$K_{\tau }(r_2)>K_{\tau }(r_1)\text{for}{r}_2>r_1,$$

but it does not follow that:

$$E_{\tau }(r_2)>E_{\tau }(r_1).$$

The Parametric Memory Law

Xu et al.'s How LoRA Remembers? A Parametric Memory Law for LLM Finetuning proposes an empirical power law:

$$\Delta \mathcal{L}(r,\ell )=C{r}^{\alpha }{\ell }^{-\beta }+b,$$

where:

• $\Delta \mathcal{L}$ is the reduction in answer-token loss;
• $r$ is LoRA rank;
• $\ell$ is target sequence length or knowledge load;
• $\alpha >0$ is a capacity exponent;
• $\beta >0$ is a length-penalty exponent;
• $C$ and $b$ depend on the model, data distribution, placement, and training protocol. [S2]

Under fixed module placement, $P_{\mathrm{LoRA}}\propto r$, giving:

$$\Delta \mathcal{L}\propto P_{\mathrm{LoRA}}^{\alpha }{\ell }^{-\beta }.$$

The important interpretation is not the exact coefficients. It is the observed sublinear power-law trade-off:

• More adapter parameters reduce memorization loss.
• Longer or denser targets are harder to store.
• Doubling rank does not generally double reliable memory.
• Exponents are empirical properties of a specific experimental regime, not universal constants.

Loss Is Not Equivalent to Exact Recall

A model may achieve low average cross-entropy while still failing exact generation.

For a target sequence:

$$y=(y_1,\dots ,y_T),$$

successful autoregressive recall requires the correct token to win at every position under the generated prefix.

Under greedy decoding, the condition:

$$p(y_t\mid y_{<t},x)>0.5$$

is sufficient for $y_t$ to be the unique most probable token at position $t$.

If one early target token remains below the necessary dominance threshold:

1. A competing token may be selected.
2. The generated prefix deviates from the training prefix.
3. Later conditional distributions change.
4. Subsequent errors cascade.
5. Sequence-level exact match becomes zero.

Thus:

$$\text{low mean loss}\text{centernot}⟹\text{exact recall}.$$

This is why capacity experiments should report:

• answer-token cross-entropy;
• token-level accuracy;
• sequence exact match;
• paraphrase-level semantic accuracy.

Capacity Has a Threshold Region, Not a Sharp Universal Number

A practical memory curve commonly contains three regimes:

Regime	Behaviour
Under-capacity	High recall; additional data may improve optimization
Transition region	Seed-sensitive, prompt-sensitive, partial recall
Over-capacity	Rapid loss of exact recall and increased interference

The “capacity” of rank $r$ therefore depends on the chosen reliability criterion:

$$K_{0.99}(r)<K_{0.90}(r)<K_{0.50}(r).$$

Reporting one capacity number without its threshold is incomplete.

Why There Is No Universal “Facts per Weight” Constant

Information-Theoretic Interpretation

Raw Storage Bits Are Not Learned Information

A BF16 parameter physically occupies $16$ bits, but it does not follow that each learned parameter stores $16$ useful bits.

Reasons include:

• correlated parameters;
• factorization redundancy;
• optimizer noise;
• flat directions;
• finite effective precision;
• error-correction redundancy;
• unused nominal rank;
• robustness across prompt variations.

The useful information per parameter is therefore much smaller than the physical representation size.

Dense-Model Bits-per-Parameter Estimates

Two broader dense-transformer studies provide useful—but non-LoRA-specific—reference points:

• Allen-Zhu and Li's Knowledge Capacity Scaling Laws reports approximately 2 factual knowledge bits per parameter in controlled tuple-learning experiments.
• Morris et al.'s How Much Do Language Models Memorize? estimates approximately 3.6 memorization bits per parameter after experimentally suppressing generalization. [S3]

These values must not be directly presented as LoRA capacity constants because those studies concern models trained more broadly rather than adapters attached to frozen backbones.

They are best treated as:

An order-of-magnitude sanity envelope for neural parametric memory, not a prediction of LoRA recall.

A Conditional Entropy Bound

A rough upper-bound framework is:

$$N_{\mathrm{facts}}\lesssim \frac{\eta P_{\mathrm{eff}}}{\overline{H(Z_i\mid {\Theta }_0,K_i)}},$$

where:

• $\eta$ is useful learned information per effective parameter;
• $P_{\mathrm{eff}}$ is the number of functionally useful adapter degrees of freedom;
• $K_i$ is the query or key for fact $i$;
• $Z_i$ is the associated answer;
• $H(Z_i\mid {\Theta }_0,K_i)$ is the residual entropy of that answer given the base model and key.

This expression clarifies why “one fact” is not a fixed unit.

Examples:

• A binary attribute requires at most one irreducible bit when the entity is already known.
• One uniformly random 10-digit number contains:

$${\log }_2(10^{10})\approx 33.2\text{ bits},$$

before accounting for the name–number association and robust retrieval.

• A redundant natural-language fact may be highly compressible.
• A novel algorithm cannot be adequately characterized as a short fixed number of factual bits.

Illustration, Not a Capacity Prediction

Suppose an adapter contains $10^6$ effective parameters. Applying the dense-model range of $2$–$3.6$ bits per parameter gives a purely theoretical envelope of:

$$2.0\times 10^6\text{ to }3.6\times 10^6\text{ bits}.$$

For random 10-digit values alone:

$$N\lesssim \frac{2.0\text{M–}3.6\text{M}}{33.2}\approx 60,000\text{–}108,000.$$

This estimate is unrealistically optimistic because it excludes:

• storage of the keys;
• binding each key to its value;
• prompt paraphrases;
• output syntax;
• resistance to interference;
• confidence calibration;
• error correction;
• sequence-decoding failures;
• conflict with base-model priors.

Semantic Compression Changes Apparent Capacity

Consider three datasets containing the same number of textual tokens:

Dataset	Compressibility	Expected memorization burden
Repeated template with predictable values	High	Low
Natural facts sharing entity and relation structure	Medium	Medium
Random keys mapped to random strings	Minimal	High

A model can store more apparent “facts” when those facts share structure because it can encode the rule plus exceptions rather than every item independently.

Therefore:

$$\text{token count}\ne \text{information content}\ne \text{retrievable knowledge}.$$

Controlled random associations are useful for capacity measurement precisely because they minimize semantic compression and generalization.

Factors That Alter Knowledge Capacity at Fixed Parameter Count

Capacity Is a Property of the Whole Training System

Module Placement

Two adapters with the same number of parameters can have different capacity because they intervene at different computational locations.

Placement	Potential role in factual memory
Early attention	Entity and lexical feature selection
Middle attention	Contextual binding and relation tracking
Late attention	Query-conditioned retrieval
MLP projections	Storage and transformation of feature associations
Embeddings	Representation of newly introduced tokens
LM head	Direct alteration of output-token preferences

The parameter count:

$$P_{\mathrm{LoRA}}=\sum _{\ell }{r}_{\ell }(d_{\mathrm{in},\ell }+d_{\mathrm{out},\ell })$$

does not record whether those parameters are located in useful modules.

A better capacity description includes:

$$\mathcal{C}=f(P,\{r_{\ell }\},\mathcal{S},{\Theta }_0,\mathcal{D},\mathcal{O}),$$

where $\mathcal{O}$ denotes the optimization procedure.

Nominal Rank Versus Used Rank

Let the singular values of the learned update be:

$${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_r.$$

An energy-based effective rank can be defined as:

$$r_{\mathrm{eff}}(\rho )=\min \left\{k:\frac{{\sum }_{i=1}^k{\sigma }_i^2}{{\sum }_{i=1}^r{\sigma }_i^2}\ge \rho \right\}.$$

If $r=128$ but $r_{\mathrm{eff}}(0.95)=12$, most nominal directions contribute little update energy.

Possible causes:

• data do not require the additional directions;
• optimization fails to activate them;
• regularization suppresses them;
• module placement is inappropriate;
• the task itself has low intrinsic dimension.

Training Data Format

Back et al. find that, for knowledge internalization, structured synthetic supervision outperforms raw-text training. Their observed ordering was approximately:

$$\text{QA}>\text{Summary}>\text{Rewrite}>\text{Raw text}.$$

QA also produced the largest gain per training token, although part of this benefit may arise from alignment between the training and evaluation formats. [S4]

The implication is that capacity depends on supervision density:

• Raw text contains many tokens unrelated to the fact that must be retrieved.
• Summaries compress high-value content.
• QA pairs expose an explicit key–value relationship.
• Multiple paraphrases teach prompt invariance.
• Negative examples help define relation boundaries.

Number of Exposures Versus Number of Unique Facts

Increasing training tokens can mean:

1. Adding new facts.
2. Repeating existing facts.
3. Adding paraphrases of existing facts.
4. Adding reasoning paths connecting existing facts.

These consume parameter and compute budgets differently.

Define:

$$N_{\mathrm{unique}}\ne N_{\mathrm{examples}}\ne N_{\mathrm{tokens}}.$$

A capacity study must report all three.

Repetition can improve exact recall without increasing stored information. Paraphrases can improve retrieval robustness while increasing training cost. New facts increase information load directly.

Conflict With Base Knowledge

Learning a novel association and revising an existing belief are different optimization problems.

For a counterfactual target $z_{\mathrm{new}}$ opposing a strong base answer $z_{\mathrm{old}}$, the adapter must satisfy both:

$$p_{{\Theta }_0+\Delta \Theta }(z_{\mathrm{new}}\mid x)\uparrow$$

and:

$$p_{{\Theta }_0+\Delta \Theta }(z_{\mathrm{old}}\mid x)\downarrow .$$

The second requirement may consume substantial adapter capacity because the base model continually contributes the old belief through the frozen path.

Potential outcomes include:

• successful replacement;
• context-dependent alternation;
• answer blending;
• overgeneralized replacement;
• regression to the original answer under paraphrasing.

Base-Model Scale and Quality

A stronger base model may reduce the residual information required for structured knowledge because it already supplies:

• entity representations;
• relation schemas;
• linguistic variation;
• reasoning primitives;
• output formatting.

However, empirical results do not support a simple monotonic law in which larger backbones always produce proportionally better LoRA memory. Improvements can be irregular because the adapter remains the principal writable component while the backbone acts as frozen infrastructure. [S4]

Capacity, Generalization, and Forgetting

The Plasticity–Stability Trade-off

Learning More Usually Permits Forgetting More

Biderman et al.'s LoRA Learns Less and Forgets Less reports that low-rank LoRA underperforms full fine-tuning in demanding code and mathematics adaptation, especially under continued pretraining. Full fine-tuning learned update matrices with substantially higher effective rank than standard LoRA configurations. [S5]

This supports the following continuum:

Method	Plasticity	Retention of base behaviour
Very-low-rank LoRA	Low	High
Higher-rank/all-linear LoRA	Medium	Medium–high
Partial full-rank tuning	High	Medium
Full fine-tuning	Highest	Lowest without constraints

The design principle is:

$$\text{capacity to acquire}\leftrightarrow \text{capacity to overwrite}.$$

The low-rank constraint is simultaneously:

• a memory bottleneck;
• a regularizer;
• a protection against uncontrolled drift;
• a barrier to major capability acquisition.

New-Knowledge Accuracy Can Hide Global Damage

Pletenev et al.'s How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM? finds that mixing known and new facts can improve knowledge injection, but external QA performance can still decline. Biased training data may also cause regression toward a small number of overrepresented answers. [S6]

A complete evaluation therefore requires:

$$\text{utility}=\text{new knowledge}+\text{retained knowledge}+\text{locality}+\text{general capability}.$$

Do not evaluate only the newly inserted facts.

Knowledge-Update Metrics

Metric	Question answered
Reliability	Does the model answer the direct training query correctly?
Generalization	Does the answer survive paraphrasing?
Portability	Can the fact participate in related reasoning?
Locality	Are unrelated outputs unchanged?
Retention	Are previously known facts preserved?
Specificity	Is the update restricted to the intended entity/relation?
Calibration	Is confidence appropriate?
Compositionality	Can multiple learned facts be combined?
Reversibility	Can the update be removed cleanly by disabling the adapter?

Scaling Beyond a Single LoRA

Modular Parametric Memory

Multiple Small LoRAs Versus One Large LoRA

Under a fixed total parameter budget, partitioning knowledge across multiple small LoRAs can outperform one monolithic adapter if the correct module is always selected. [S7]

Suppose:

$$P_{\mathrm{total}}=\sum _{j=1}^M{P}_j.$$

A modular system stores disjoint knowledge partitions:

$$\mathcal{K}=\bigcup _{j=1}^M{\mathcal{K}}_j,{\mathcal{K}}_i\cap {\mathcal{K}}_j\approx \varnothing .$$

Potential benefit:

• each adapter remains below its saturation threshold;
• unrelated facts do not compete within one low-rank update;
• modules can be added or removed independently;
• updates are easier to version.

Under oracle routing:

$$j^{\ast }=\mathrm{Router}(x),$$

and only ${\varphi }_{j^{\ast }}$ is activated.

This converts a single-adapter capacity problem into a routing problem.

Routing Becomes the New Bottleneck

Practical embedding-based routing can perform substantially worse than oracle routing and may even underperform a single broad LoRA. [S7]

Error decomposition:

$$P(\text{correct})=P(\text{correct}\mid \text{right module})P(\text{right module})+P(\text{correct}\mid \text{wrong module})P(\text{wrong module}).$$

Highly specialized modules make:

$$P(\text{correct}\mid \text{wrong module})$$

particularly small.

The modular architecture therefore succeeds only if:

• routing recall is high;
• knowledge partitions are semantically separable;
• multi-hop queries can activate all relevant modules;
• routing latency remains acceptable;
• module identity is stable under paraphrasing.

Merging Is Not Equivalent to Routing

For adapters $\Delta W_1,\dots ,\Delta W_M$, direct merging produces:

$$\Delta W_{\mathrm{merge}}=\sum _{j=1}^M{\lambda }_j\Delta W_j.$$

This changes every query, even when only one adapter is relevant.

Strategy	Advantage	Main failure
Top-1 routing	Minimal interference	Catastrophic misrouting
Top-$k$ routing	Better recall	More irrelevant adapters
Linear merge	Simple	Directional cancellation/interference
Concatenation	Preserves summed updates	Effective rank grows as ${\sum }_j{r}_j$
Learned mixture	Flexible	Requires additional training/router
Single monolithic LoRA	No routing error	Saturation and internal interference

The set of matrices of rank at most $r$ is not closed under addition:

$$\mathrm{rank}\left(\Delta W_1+\Delta W_2\right)\le \mathrm{rank}(\Delta W_1)+\mathrm{rank}(\Delta W_2).$$

The sum of two rank-$r$ adapters may require rank up to $2r$ to represent exactly.

A Rigorous Experimental Protocol for Measuring LoRA Capacity

Capacity Should Be Measured, Not Assumed

Research Question Definition

A defensible experiment should specify which capacity is being estimated:

• arbitrary key–value memorization;
• counterfactual factual updating;
• natural-document internalization;
• relational generalization;
• procedural capability acquisition;
• domain continued pretraining.

These are different dependent variables and should not be collapsed into one “knowledge capacity” score.

Controlled Dataset Construction

Use nested datasets:

$${\mathcal{D}}_{K_1}\subset {\mathcal{D}}_{K_2}\subset \cdots \subset {\mathcal{D}}_{K_n},$$

where $K_i$ is the information load.

For each load, report:

• unique facts;
• answer tokens;
• total training tokens;
• estimated entropy;
• number of paraphrases;
• number of exposures;
• degree of conflict with base knowledge.

Include:

• random associations to estimate raw memory;
• natural facts to estimate compressible memory;
• counterfactual facts to estimate revision cost;
• unrelated controls to estimate locality.

Rank and Placement Sweep

At minimum, sweep:

$$r\in \{2,4,8,16,32,64,128,256\}.$$

Compare:

• attention-only;
• MLP-only;
• all-linear;
• equal total-parameter configurations;
• uniform-rank versus non-uniform-rank allocations.

The important control is to separate:

$$\text{effect of rank}$$

from:

$$\text{effect of total parameters}$$

and:

$$\text{effect of placement}.$$

Optimization Controls

For each configuration:

• tune learning rate rather than reusing one value for every rank;
• train long enough to distinguish capacity failure from undertraining;
• report both fixed-step and convergence-matched results;
• use several random seeds;
• track gradient norms for $A$ and $B$ separately;
• inspect singular values of $BA$;
• test LoRA+ or improved initialization when optimization failure is suspected.

A rank may appear insufficient merely because its factorization was poorly optimized.

Evaluation Grid

Axis	Required test
Direct recall	Original question
Surface robustness	Several paraphrases
Context robustness	Distractor context
Relation reversal	Ask inverse relation where valid
Compositional use	Multi-hop question
Locality	Neighbouring and unrelated facts
Retention	Base benchmark before/after
Exactness	Exact match and token accuracy
Calibration	Correct-answer probability
Prompt sensitivity	Multiple templates

Estimating the Saturation Curve

For each rank, estimate:

$$K_{\tau }(r)=\max \{K:\mathrm{Score}(r,K)\ge \tau \}.$$

Then fit, where justified:

$$\log K_{\tau }=a+\gamma \log P_{\mathrm{LoRA}}+\epsilon .$$

Interpretation:

• $\gamma \approx 1$: approximately linear capacity growth.
• $0<\gamma <1$: diminishing returns.
• $\gamma >1$: possible reuse or compression effects.
• Unstable $\gamma$: regime change, optimization failure, or unsuitable capacity metric.

Report confidence intervals across seeds rather than only the best run.

Separating Storage From Retrieval Failure

When a fact is answered incorrectly, the failure may occur because:

1. It was never encoded.
2. It was encoded but the prompt did not retrieve it.
3. The right concept was retrieved but the exact value was not.
4. A competing base-model answer dominated.
5. Autoregressive decoding failed after an early token error.
6. Another learned fact interfered.

Useful probes include:

• teacher-forced target-token probabilities;
• representation similarity across paraphrases;
• logit-lens inspection;
• activation patching;
• nearest-neighbour analysis of hidden states;
• singular-vector analysis of $\Delta W$.

This connects LoRA capacity research to Mechanistic Interpretability and Associative Memory.

Design Rules for Knowledge-Bearing LoRAs

Practical Synthesis

Capacity Provisioning Rules

• Do not choose rank from dataset row count alone: estimate unique information and conflict with the base model.
• Sweep rank on nested knowledge loads: observe the empirical saturation point.
• Prefer the smallest rank satisfying the target reliability threshold: it often maximizes efficiency and retention.
• Increase placement breadth before extreme rank when the current modules may be the bottleneck.
• Inspect effective rank: unused nominal dimensions do not provide useful capacity.
• Train on retrieval-aligned formulations: direct QA plus paraphrases usually outperforms raw exposure alone.
• Include old knowledge and locality controls when updating an existing model belief.
• Use modular LoRAs only when routing accuracy can be validated.
• Retain RAG for precise, changing, auditable information.
• Treat full fine-tuning as a capability-acquisition option, not merely a larger LoRA.

Warning Signs of Capacity Overflow

Possible symptoms:

• training loss continues falling while exact match stagnates;
• performance falls as more facts are added;
• frequent confusion between entities sharing the same relation;
• regression to high-frequency answers;
• strong dependence on the original prompt template;
• newly learned facts disappear under distractor context;
• larger rank improves training recall but not paraphrase recall;
• unrelated base capabilities deteriorate;
• effective rank reaches the nominal-rank ceiling;
• different seeds produce sharply different results.

Decision Table

Observed failure	Likely cause	Appropriate response
Poor train recall at all ranks	Data or optimization problem	Improve supervision and optimization
Recall improves monotonically with rank	Capacity-limited	Increase rank or adapted modules
Train recall high, paraphrase recall low	Retrieval overfitting	Add paraphrases and semantic supervision
New facts learned, base facts damaged	Excessive plasticity/interference	Lower rank, mix retention data, regularize
One large adapter saturates	Internal competition	Partition into modules or use RAG
Multi-LoRA fails despite good modules	Routing failure	Improve router or use broader adapter
Counterfactual fact alternates with old fact	Strong frozen prior	Add contrastive/revision examples
Exact match fails after one token	Decoding bottleneck	Inspect token probabilities; target hard tokens
Higher rank provides no gain	Unused rank or placement bottleneck	Inspect spectrum; change modules
Need deletion, citations, or frequent updates	Parametric memory mismatch	Use Retrieval-Augmented Generation

Open Research Questions

Unresolved Theoretical and Empirical Problems

A Universal LoRA Capacity Law Does Not Yet Exist

Current power laws are empirical and regime-specific. Open questions include:

• Do capacity exponents remain stable above 8B models?
• How do MoE architectures change adapter memory?
• Does capacity scale differently for attention and MLP adapters?
• What is the relation between effective rank and factual entropy?
• Can capacity be predicted before training?
• How does quantization alter writable information capacity?
• What is the effect of optimizer precision on stored information?
• Can relational knowledge be compressed more effectively than arbitrary facts?
• How much apparent capacity comes from exploiting pre-existing latent features?
• Can one distinguish knowledge encoded in $A$, $B$, and the frozen residual stream?

Robust Knowledge Versus Verbatim Memory

Most controlled capacity experiments emphasize exact recall because it is easy to measure. However, doctoral-level analysis must distinguish:

$$\text{verbatim capacity}\ne \text{semantic capacity}\ne \text{reasoning capacity}.$$

A model that reconstructs a paragraph exactly may not answer a novel question about it. A model that cannot reconstruct it exactly may still have learned the relevant abstraction.

A complete theory will require separate scaling laws for:

• exact reconstruction;
• semantic retrieval;
• relational composition;
• procedural transfer;
• locality and forgetting.

Adapter Capacity as an Alignment Question

If safety or policy behaviour is encoded through low-rank adaptation, the same capacity constraints apply:

• low rank may make alignment economical;
• low rank may make alignment fragile;
• subsequent adapters may interfere with the safety update;
• a malicious adapter may redirect existing capabilities without learning them from scratch;
• removing or replacing the safety adapter may expose the base behaviour.

This connects LoRA memory to:

• Machine Unlearning;
• Continual Learning;
• Model Editing;
• Catastrophic Forgetting;
• Alignment Tax;
• Mechanistic Interpretability;
• Modular Neural Networks.

[S1] Rank sweeps from 2–1024, controlled knowledge loads, finite saturation, and declining parameter efficiency at high rank are reported by Back et al. [S2] The proposed power-law relationship, its 8B-model experiments, and the distinction between loss and exact recall come from Xu et al. [S3] The 2 and approximately 3.6 bits-per-parameter results are dense-transformer estimates under different controlled definitions of knowledge and memorization, not LoRA-specific constants. [S4] Structured QA and summary supervision, non-linear effects of base-model scale, and the limited reliability of simply switching LoRA variants are reported by Back et al. [S5] The learning–forgetting trade-off and the substantially higher-rank updates learned by full fine-tuning are reported by Biderman et al. [S6] The effects of mixing old and new facts, degradation on external QA, and regression toward overrepresented answers are reported by Pletenev et al. [S7] The advantage of partitioning knowledge under oracle routing—and the degradation introduced by practical routing—is reported by Back et al.

Knowledge-Capacity Review Checklist

• Explain why knowledge resides in the adapter–backbone interaction rather than in the LoRA alone.
• Distinguish raw parameters $r(m+n)$ from effective degrees of freedom $r(m+n-r)$.
• Explain why $\alpha$ changes update scale but not rank capacity.
• Define $K_{\tau }(r)$ and parameter efficiency $E_{\tau }(r)$.
• State the empirical rank-dependent saturation result.
• State the Parametric Memory Law and its limited empirical scope.
• Explain why low average loss does not guarantee exact autoregressive recall.
• Explain why dense-model bits-per-parameter estimates are not LoRA constants.
• Relate factual burden to conditional entropy $H(Z\mid {\Theta }_0,K)$.
• Distinguish unique facts, training examples, tokens, and information content.
• List reliability, generalization, portability, locality, and retention metrics.
• Explain why multiple small LoRAs help only when routing is accurate.
• Design a rank-by-knowledge-load experiment that identifies saturation.
• Diagnose whether failure comes from capacity, optimization, retrieval, or interference.