Muon vs AdamW for LoRA: When the optimizer changes, where should the learning rate move?

Switching from AdamW to Muon usually moved the best tested LoRA learning rate upward, while the rank-transfer pattern was more stable than the absolute scale.

LoRA [1] exposes three knobs that are tied together: rank $r$, scale $\alpha$, and learning rate lr. The original blog [11] studied how the best lr moves when r changes (under AdamW [12]). The answer depends on how $\alpha$ scales with rank: fixed $\alpha/r$ pushes the band downward, constant $\alpha$ keeps it flat, and $\alpha \propto \sqrt{r}$ is the theory-motivated rule for same-order reuse.

This post asks a different question:

When you swap AdamW for the Muon optimizer, does that rank→lr transfer story change?

Muon replaces AdamW’s coordinatewise second-moment normalization on matrix parameters with an orthogonalized momentum update [5] [6]. In our implementation, two-dimensional LoRA matrices use Muon and the training stack applies its match_rms_adamw rescaling; other parameter types follow the AdamW fallback. Because LoRA also scales its update by $\alpha/r$, changing the optimizer can change both the absolute learning-rate scale and the way that scale transfers across rank.

In our experiments, four patterns stand out:

• On AG News, Muon usually selects a higher grid point while reaching similar accuracy. Its best tested learning rate is higher in 23 of 24 (alpha rule, rank) cells, but the shift is not uniform and one cell moves lower.
• The broad rank-transfer pattern survives the optimizer change. Fixed $\alpha/r$ moves downward overall for both optimizers, while constant $\alpha$ and $\alpha \propto \sqrt{r}$ stay within a reusable order-of-magnitude band.
• On Qwen3-4B MATH, the two learning-rate grids are widely separated, but neither sweep identifies the true optimum. Muon’s grid winner is 1e-4 in every cell; AdamW usually selects its own lower boundary, 7e-7. The grids do not overlap, so an exact optimizer-to-optimizer ratio is not supported.
• Within the single-seed Qwen3-4B sweep, Muon’s best tested MATH500 accuracy is higher in all 16 cells. The observed differences range from 5.2 to 24.6 percentage points, conditional on the tested grids and selection procedure.

The useful object is therefore not one universal optimum or one universal multiplier between optimizers. It is a reusable learning-rate band, together with a clear record of where the sweep boundaries are.

All empirical claims below refer to the finite grid points that were actually evaluated. We use best tested learning rate for the grid argmax and reserve optimum for a value that the sweep genuinely brackets.

A small theory model gives three rank-transfer hypotheses

The easiest place to think about rank dependence is the start of training.

Following LoRA [1], write the adapter as

$$\Delta W = \frac{\alpha_r}{r} B A = s_r B A.$$

Under the standard initialization, $A$ is random and $B = 0$. The gradient with respect to $A$ is therefore zero at the first step, while the gradient with respect to $B$ is not. In an SGD-like first-step approximation, if denotes the gradient with respect to $\Delta W$, then

$$B^{(1)} \sim \eta \frac{\alpha_r}{r} G A^\top, \qquad \Delta W^{(1)} \sim \eta \left(\frac{\alpha_r}{r}\right)^2 G A^\top A.$$

For a random $A$ with fixed entrywise scale, $A^\top A$ aggregates over $r$ components and contributes an $O(r)$factor. This leaves the directional proxy

$$\eta \frac{\alpha_r^2}{r}.$$

This is not a model of full AdamW or Muon dynamics. It is only a way to ask how the LoRA parameterization changes the first effective step as rank changes.

More broadly, this reflects a general principle: we would like hyperparameters (such as the learning rate) to transfer smoothly across changes in model size or parameterization. This idea is closely related to muTransfer [2].

At the same time, this one-step analysis is only a simplification. In practice, optimizers like AdamW adapt updates differently across parameters. In particular, as emphasized by LoRA+ [4], the two factors A and B can benefit from different effective learning rates, meaning that the full optimization dynamics are richer than what this simple proxy suggests.

The three common scaling rules split pretty quickly:

• If $\alpha/r$ is fixed, then $\alpha \propto r$ and the proxy grows with rank. Higher rank should push the search toward smaller learning rates.
• If $\alpha$ is fixed, the proxy shrinks with rank. Higher rank should not force smaller learning rates.
• If $\alpha \propto \sqrt{r}$ the rank dependence cancels. This is the simplest route to reusing the same order of learning rate, and it is also the scaling singled out by rsLoRA [3].

The proxy predicts rank direction, not the absolute ratio between AdamW and Muon. Their normalization geometries differ, so that ratio has to be measured. This distinction becomes important in the Qwen3-4B sweep, where both optimizers select boundary points from non-overlapping grids.

How to read the sweeps: grid winners, not identified optima

For each (scaling rule, rank) cell, we select the evaluated learning rate with the highest reported metric. On AG News, the values are three-seed means (Muon seeds 42–44; the AdamW reference uses seeds 41–43). On Qwen3-4B MATH, the comparison is single-seed.

The optimizer-specific grids are deliberately different. AG News uses seven AdamW rates from 1e-5 to 1e-3 and ten Muon rates from 1e-5 to 3e-3. Qwen3-4B uses AdamW rates from 7e-7 to 5e-5 and Muon rates from 1e-4 to 3e-3. A grid winner at an interior point is informative about a local band. A winner at the smallest or largest tested rate is a request to extend the sweep, not evidence that the true optimum has been located.

AG News is the easiest place to see it in the data

AG News [8] is useful here because most selected points sit inside the learning-rate grids. We use the DistilBERT [7] classification setting and sweep ranks 2, 4, 8, 16, 32, 64, 128, 256.

We compare three scaling families under both optimizers: constant $\alpha$ =32, fixed $\alpha/r=1$, and $\alpha \propto \sqrt{r}$ .

Figure 1. The grid argmax for each rank and scaling rule. These are best tested rates, not continuous optima.

The three panels tell a more nuanced story than a single multiplier:

• Constant $\alpha$=32: AdamW selects 2e-4 at every rank. Muon stays in a higher but non-monotonic 3e-4–7e-4 band.
• Fixed $\alpha/r$=1: both optimizers move downward overall. AdamW goes from 5e-4 at low rank to 1e-4–2e-4 at high rank; Muon goes from 1e-3 at ranks 2 and 4 to 3e-4 at ranks 64–256, with a non-monotonic middle.
• $\alpha \propto \sqrt{r}$: AdamW is almost flat at 2e-4; Muon selects 3e-4 at five of eight ranks and remains within 3e-4–7e-4. This is same-order reuse, not exact invariance.

Across all 24 cells, Muon selects a higher learning rate in 23. When it is higher, the ratio ranges from 1.4× to 5×. The sole reversal is fixed $\alpha/r$=1 at rank 16, where Muon selects 3e-4 and AdamW selects 5e-4.

The accuracy result is deliberately less dramatic. AG News is close to saturation: the difference between the best tested Muon and AdamW accuracies ranges from −0.43 to +0.63 percentage points, with a mean of +0.11 points. On this task, the useful result is the location and shape of the reusable learning-rate band, not a performance win.

Qwen3-4B on MATH makes the differences harder to ignore

We then moved to the harder setting from the original blog:

• model: Qwen/Qwen3-4B-Instruct-2507 [9]
• train set: MATH \ MATH500 [10]
• selection and reported metric: MATH500 accuracy
• ranks: 2, 8, 32, 128
• configurations: constant $\alpha=32$, fixed $\alpha/r=1$, fixed $\alpha/r=2$, and $\alpha \propto \sqrt{r}$

The first result is about experimental resolution. Muon selects 1e-4, the smallest tested Muon rate, in all 16 cells. AdamW selects 7e-7 in 13 cells, 1e-6 in two cells, and 5e-6 in one cell. The grids do not overlap.

The observed grid winners are therefore separated by roughly one to just over two orders of magnitude, depending on the cell. But because both sweeps frequently select a lower-boundary point, the true optima and their ratio remain unresolved. Saying that Muon’s optimum is exactly 100× larger would overstate what this grid can identify.

Figure 2. Best tested single-seed MATH500 accuracy for each rank and scaling configuration.

Within these finite, single-seed sweeps, the best tested Muon run is higher in all 16 (configuration, rank) cells. The smallest difference is 5.2 percentage points ($\alpha \propto \sqrt{r}$ rank 2: 0.714 vs 0.662). The largest is 24.6 points ($\alpha/r$=2, rank 128: 0.674 vs 0.428).

At rank 128, the best tested AdamW accuracy spans 0.428–0.606 across the four configurations, while Muon spans 0.674–0.734. In this slice, Muon appears less sensitive to the alpha rule. That is a descriptive result from this sweep, not yet a general mechanism claim.

The performance comparison also needs the same restraint as the learning-rate comparison. The runs use different optimizer-specific grids, MATH500 is both the selection metric and the reported metric, and the sweep has one seed. The result is strong enough to motivate replication, but not to establish a causal advantage from orthogonalization alone.

What we would do in practice

If we are changing LoRA rank in a real sweep with Muon, we would not begin with a universal AdamW-to-Muon multiplier. We would begin with the scaling rule, then use a short calibration sweep to locate the optimizer’s working band.

A simple playbook (Muon edition):

Start with a transferable alpha rule

Start with $\alpha \propto \sqrt{r}$.

The early-step proxy cancels the explicit rank dependence under this rule, and the AG News sweep keeps the selected Muon rate within 3e-4–7e-4. This makes it a sensible default for same-order reuse, not a guarantee that one exact rate will transfer.

Recalibrate when the optimizer changes

On AG News, Muon usually selects a higher rate, but the shift varies by rule and rank. On Qwen3-4B, the two tested grids are far apart, yet both optimizers often select their lower boundary. The practical response is to run a small logarithmic pilot grid and extend it whenever the winner lands at an edge.

Under fixed alpha/r, search lower as rank grows

Both AG News trajectories move downward overall under fixed $\alpha/r$=1. For Muon, the best tested rate falls from 1e-3 at ranks 2 and 4 to 3e-4 at ranks 64–256. This is the clearest case where rank should change the center of the search grid.

Treat a boundary winner as an unfinished measurement

The Qwen3-4B sweep does not locate Muon’s optimum because every cell selects the minimum tested rate. Before turning that pattern into a deployment recommendation, extend the grid below 1e-4 and replicate the comparison with additional seeds.

The useful object is a reusable band, not one magic learning rate

People often frame transfer as a yes-or-no question: does the same best learning rate work everywhere?

We do not think that is the right level to look at it.

The more useful object is the reusable learning-rate band.

Different optimizers can place that band at different absolute scales, but the alpha rule still shapes how it moves across rank:

• fixed $\alpha/r$ pushes the search downward overall for both optimizers on AG News
• constant alpha gives AdamW a flat selected rate and Muon a higher, non-monotonic same-order band
• $\alpha \propto \sqrt{r}$ preserves same-order reuse for both, while leaving the absolute optimizer-to-optimizer scale task dependent

The practical conclusion is simple: do not copy an AdamW learning rate unchanged, and do not multiply it by a universal constant. Transfer the shape of the search, then recalibrate its center.

Limitations

• The theory section is an early-training, SGD-like scaling argument, not a theorem about AdamW or Muon.
• AG News is near saturation. Its three-seed comparison is useful for learning-rate location, but the best-accuracy differences remain within about ±0.7 percentage points.
• The Qwen3-4B comparison is single-seed, uses MATH500 for both model selection and reporting, and frequently selects a grid boundary. It does not identify the continuous optimum.
• The AdamW and Muon grids differ by design. The performance comparison is therefore conditional on the tested search spaces.
• The implementation is hybrid and stack-specific: two-dimensional LoRA matrices use Muon, while other parameter types use the AdamW fallback and the stack’s match_rms_adamw rescaling.

References

[1] LoRA: Low-Rank Adaptation of Large Language Models (Hu et al, 2021)

[2] Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (Yang et al, 2022)

[3] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (Kalajdzievski, 2023)

[4] LoRA+: Efficient Low Rank Adaptation of Large Models (Hayou et al, 2024)

[5] Muon: An optimizer for hidden layers in neural networks (Jordan, 2024)

[6] Muon is Scalable for LLM Training (Liu et al, 2025)

[7] DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (Sanh et al, 2020)

[8] Character-level Convolutional Networks for Text Classification (Zhang et al, 2015)

[9] Qwen3 Technical Report (Yang et al, 2025)

[10] Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al, 2021)

[11] Triquetra for LoRA’s entangled knobs: When LoRA rank changes, how should learning rate move? (Mind Lab, 2026)

[12] Decoupled Weight Decay Regularization (Loshchilov and Hutter, 2019)

Author

Mind Lab

Core Contributors

Wenbin Wang, Kieran Liu, Andrew Chen, Pony Ma

Team

Asher Cai, Song Cao, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Jun Gao, Pyke Han, Nolan Ho, Mutian Hong, Charles Huang, Fancy Kong, Andrew Lei, Lucian Li, Ray Li, Fan Lin, Kieran Liu, Logan Liu, Neo Liu, Xiang Liu, Yuxin Lu, Pony Ma, Cole Qiao, Vince Qu, Vincent Wang, Bo Wu, Chengdong Xu, Rio Yang, Regis Ye, Yihang Zeng, Di Zhang, Jiahao Zheng, Adrian Zhou, Yuhua Zhou, Murphy Zhuang and Mindverse Team

Names are listed alphabetically within team.

Citation

Please cite this work using the BibTeX citation:

@misc{smithli2026dcp2dreshard,
  author = {Wenbin Wang and Kieran Liu and Andrew Chen and Pony Ma and {Mind Lab}},
  title = {Muon vs AdamW for LoRA: When the optimizer changes, where should the learning rate move?},
  year = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note = {https://macaron.im/mindlab/research/muon-vs-adamw-for-lora-when-the-optimizer-changes-where-should-the-learning-rate-move}
}