The Best Optimizer Depends on Batch Size

† Equal contribution.

Read the paper

Larger batches can speed up training, but often require a different learning rate and other hyperparameters.1 Scaling rules tell us how to adjust these settings without another search. But different theories recommend different rules for the same optimizer.

Q1Can a hyperparameter scaling rule consistently work in different training settings?

Batch size may also change which optimizer works best. Larger batches reduce gradient noise, which can affect optimizers differently. However, standard optimization benchmarks like AlgoPerf and NanoGPT rank optimizers according to experiments run at a fixed batch size. We compare optimizers across batch sizes and carefully retune each one at every batch size.

Q2Do optimizer rankings persist across batch sizes after extensive retuning?

Deriving and testing scaling rules.

We use Muon to test how well these competing scaling rules work across tasks and batch sizes.

Muon optimizer

Momentum
Mt=μMt−1+(1−μ)Gt
Update direction
Ut=Orth(μMt+(1−μ)Gt)
Parameter update
Wt+1=(1−ηMλM)Wt−ηMUt
Wt
Matrix weights
Gt
Minibatch gradient
μ
Momentum retention
ηM
Learning rate
λM
Weight decay
Mt
Momentum
Orth
Approximate orthogonalization using Newton–Schulz.

The subscript M denotes the matrix-parameter group.

In these experiments, Muon updates matrix weights and AdamW updates auxiliary parameters.2 The equations below show how each rule adjusts Muon’s hyperparameters when the batch size grows by a factor of κ. Primes denote their values at the new batch size.

Matching an SDE

Linear learning rate.

ηM′=κηM,μ′=μκ,λM′=λM

We adapt an SDE-matching framework for Adam and RMSprop to Muon.3 The derivation holds the gradient–momentum mixing coefficient fixed, but Muon ties it to momentum, so the match is not exact.

Minimizing a bound

Square-root learning rate.

ηM′=κηM,μ′=μ,λM′=λM

This approach balances two terms in a theoretical convergence bound: one for initialization and one for loss smoothness.4 The square-root rule applies until the bound’s effective step-size cap is reached.

Power Lines

Fixed learning rate.

ηM′=ηM,μ′=μ,λM′=κλM

Power Lines keeps AdamW’s nominal parameter-averaging timescale fixed as batch size changes at a fixed token budget.5 We apply the same identity to Muon while holding momentum fixed.

We tune this configuration at a reference batch size for language modeling and image classification, then increase the batch size while keeping each task’s training budget fixed. We compare hundreds of scaling rules, including the candidates above and other combinations of learning rate, momentum, and weight-decay adjustments.6

Takeaway: The best scaling rule depends on both the task and target batch size.

Scaling-rule search.

Validation lossLrulenats

Batch sizeB′tokens / update

Selected ruleBest tunedShared baselineOther rules

Loss rank
Protocol

An additional sweep at 1M and 2M tokens found that scaling auxiliary AdamW weight decay offers little benefit. The language-model grid therefore holds that coordinate fixed.

Each curve is a complete rule. They share one reference configuration at 128K tokens or 256 images per update; this point is excluded from target-batch ranks and mean gaps. The chart shows transfers to four larger batches: 864 language-model runs or 2,592 CIFAR-5M runs, one seed per setting. CIFAR starts at the shared 256-image reference and transfers to 512, 1K, 2K, and 4K. The 64- and 128-image scale-down experiments remain in the complete dataset (3,888 CIFAR runs), but are excluded from this scale-up comparison and its rankings. Language-model loss is terminal validation cross-entropy after approximately 1.7B tokens; CIFAR-5M loss averages the final ten evaluations of one pass over 5M training images. Loss gap subtracts the lowest recorded loss at the same batch across the original grid and available full-retuning medians. Both loss views use linear y axes and retain every measured endpoint in this scale-up comparison. Detail fits the selected rule and its baseline across all batches. Loss rank orders the tested rules by validation loss at the current batch: 1 is best, and equal losses share a rank. The denominator is the number of rules in the selected dataset; the rank updates when the rule or batch changes. Full range shows every endpoint in the main plot. Curves outside the detail window are clipped, without changing their values. Selecting a rule fits its full curve; Full range restores all runs to the main plot.

“Best common rule” minimizes mean regret relative to the best tested grid rule across the four larger target batches. “Best at this batch” minimizes only the selected batch’s loss. Bound, SDE, and Power Lines presets show the completion of each Muon prescription with the lowest mean grid regret across the four larger target batches; the derivations alone do not specify every auxiliary AdamW choice. Rule rankings and common-rule selection apply the paper’s mean grid-regret criterion to these four larger targets. For CIFAR this is a different cohort from the paper’s six-target ranking, which also includes scaling down. Displayed gaps instead use the lowest recorded grid or retuning loss, and their mean weights the four larger targets equally. The language-model baseline uses retuned medians at 512K–2M and the paper’s best-grid proxy at 256K. CIFAR-5M baselines are three-seed retuned medians at 512–4096; when a single-seed grid run lies below a retuned median, that grid loss becomes the displayed reference.

For retention-preserving coefficients, CIFAR-5M uses μ′=μκ,β1′=β1κ,β2′=β2κ. Language modeling uses reference updates divided by target updates instead of the batch ratio; at 2M, this is 13,000 / 813. LR/WD choices use the batch ratio directly.

Optimizer rankings across batch sizes.

We compare Adam, Lion, Muon, SOAP, and Shampoo in language-model pretraining, retuning each optimizer at every batch size from 128K to 2M tokens. All runs use the same total training budget. We test both weight decay, which shrinks weights, and HyperBall, which keeps each hidden matrix’s norm fixed.

Under HyperBall, Muon outperforms Shampoo at 128K tokens per update, while Shampoo outperforms Muon at 2M.7

Takeaway: The best optimizer at one batch size may lose at another, even after retuning.

Optimizer rankings.

Validation lossLnats
Validation loss versus batch size

Batch sizeB′tokens / update

At 128K tokens / updateLoss

Pairwise comparison.

Loss differencenats

Protocol

The shaded band marks ±0.002 nats.

Curvature and gradient noise.

We use a quadratic loss with noisy gradients to understand why optimizer rankings can change with batch size.8 We compare SGD with Newton’s method, which divides each direction’s gradient by its curvature.

Noisy quadratic model

Loss
L(w)=12(w12+h2w22)
Minibatch gradient
g^i=hiwi+ciBzi
SGD
wi←wi−ηg^i
Newton
wi←wi−ηhig^i
wi
Position in direction i
hi
Curvature in that direction
ci
Single-example gradient-noise variance
B
Batch size
η
Learning rate
zi
Independent standard Gaussian noise, redrawn each step

In our setting, h1=1 and c2=0: the first direction is flat and noisy; the second is sharp and noiseless.

We choose each method’s learning rate to give the lowest average loss after 4,096 samples. Larger batches give us less noisy gradients, but fewer updates.

The map shows which optimizer does better on average. The animation follows one run, so its outcome can differ. Our theorem shows that rankings can reverse even after optimal tuning.9

Takeaway: Newton can reduce bias (remaining initial error) at the cost of higher variance (fluctuations from noisy gradients). Larger batches reduce the variance, which can make Newton beat SGD.

Change the geometry. Does the winner change?

Noisier / more stepsCleaner / fewer steps
Rounder valleySteeper walls
Start with error in
0 / 4,096 samples

Optimizer paths

SGD wins
1.0×
w1FlatNoisyw2SharpNoiseless
SGDNewtonw1 Flat · noisyw2 Sharp · noiseless
Live lossLog scale
SGDNewton

Winner map

The columns show batch sizes, and the rows show noise levels.

SGDNewtonClose or tiedDeeper color = larger loss ratio
Expected loss · 4,096 samples

The bars break the loss into two parts:

expected loss = remaining initial error + loss from gradient noise

Solid color marks the initial error that remains. Stripes mark the extra loss caused by noisy gradients.

Protocol

The objective is L(w)=12(w12+h2w22). Gradients are g^i=hiwi+ciBzi, with independent standard Gaussian noise at every step. Here h1=1, c1 is the noise slider, and c2=0. The sample budget is T=4,096 and the number of updates is K=TB. Both methods start at the same point and use the same underlying noise draws.

SGD uses wi←wi−ηg^i. Newton uses wi←wi−ηhig^i. The learning rate is constant within each run. The canvas shows original parameter coordinates.

We tune each optimizer’s learning rate within its stable range to minimize expected loss after 4,096 samples. The animation shows individual runs, which can fluctuate near the minimum because of gradient noise.

For every sufficiently large sample budget, there exists a two-dimensional NQM on which optimally tuned SGD outperforms optimally tuned Newton at batch 1, but the ranking reverses when the whole budget forms one batch. The theorem holds both without momentum and with optimally tuned momentum. The reversal comes from a bias–variance tradeoff: Newton’s preconditioning can reduce initialization error faster at the cost of greater variance from gradient noise. Larger batches reduce the noise contribution, changing the balance between the two optimizers. We next examine how this tradeoff varies across directions and affects their learning rate scaling rules.

Directional scaling rules.

We study SignSGD on a one-dimensional noisy quadratic, where the sign of momentum determines the update direction.

SignSGD and curvature-to-noise ratio

Momentum
mt+1=μmt+(1−μ)gt
Parameter update
wt+1=wt−ηsign(mt+1)
Curvature-to-noise ratio
CNR=h2c
wt
Parameter value
gt
Noisy minibatch gradient
mt
Momentum average, initialized to zero
μ
Momentum retention, fixed at 0.9
η
Learning rate
h
Curvature
c
Single-example gradient-noise variance

The sign tells us which way to move, and the learning rate sets the step size. Higher CNR means less noise relative to squared curvature.

In the binoculars below, the valley’s shape represents curvature and blur represents gradient noise. At the same batch size and distance from the minimum, higher CNR makes the downhill direction easier to identify.

A children’s-book illustration of one pair of binoculars: the left lens, labeled High CNR, shows a clearly focused steep green valley; the right lens, labeled Low CNR, shows a flat plain blurred like nearsighted vision.

We vary the noise and retune the learning rate at each batch size, keeping other settings fixed. As CNR increases, the tuned rates shift from roughly square-root toward linear scaling.10

Takeaway: Even for the same optimizer, the best scaling rule depends on the landscape and the gradient noise.

SignSGD learning rate scaling.

Tuned learning rate ratioRelative to reference batchη(κB)η(B)

Batch ratioκ=B′B

Linear κSquare-root κTuned
Protocol

In the noisy quadratic setting, ideal Muon orthogonalization on a 1 × 1 block is the sign operation, motivating this SignSGD comparison. The ratio is CNR=h2c.

Momentum is fixed at μ=0.9. Curvature is h=1, single-example noise variance is c=1/CNR, initialization is w0∼N(0,1), and initial momentum is zero. The recursion applies the sign of momentum before each parameter update, without weight decay. For each batch, the search minimizes expected terminal loss after the fixed 4,096-sample budget.

The paper uses 121 broad logarithmic learning rate candidates with 512 paths, refines three separated neighborhoods, selects with 2,048 independent paths, and evaluates on eight held-out groups of 1,024 paths. The fitted exponent is an ordinary least-squares slope of log learning rate against log batch over the nine tested batches. It summarizes those endpoints rather than asserting a universal power law.

The playable adds 325 independently tuned toy-model points, giving 22 CNR conditions and 16 batches. The 27 paper measurements and their fitted exponents remain unchanged. Each slider position selects an actually tuned CNR; lines connect measured points. Additional batches use floor(4,096/B′) updates, with actual sample counts recorded in the dataset. Supplemental tuning follows the same search and independent selection protocol, then evaluates the chosen rate on eight held-out groups.

Takeaway: One learning rate scaling rule cannot preserve movement in both directions as batch size grows.

Can one exponent preserve both directions?

Here, movement means SignSGD’s average progress toward the minimum per sample, measured at a fixed position. At 1×, that progress matches the batch-1 reference.11

new learning rate = reference learning rate × batch ratioexponent

Movement on a quadratic

Expected displacement per sampleRelative to batch 1

Batch ratioκ=B′B

High CNR (1) Low CNR (0.001)Batch-1 reference

The two SignSGD curves respond differently because:

  1. When noise often flips the sign, larger batches help SignSGD choose the right direction. Roughly square-root learning rate scaling keeps progress per sample similar.
  2. When the sign is already reliable, larger batches mainly mean fewer updates. Roughly linear learning rate scaling makes up for those missing steps.

The previous chart instead chooses learning rates based on loss at the end of a full run.

Protocol

Throughout the blog, B is a batch size before scaling, B′ the scaled batch, and κ=B′B their ratio. η is a learning rate, μ a momentum coefficient, and α a learning rate exponent. The local curves hold both coordinates at w=1 and assume stationary momentum with μ=0.9.

Assume independent Gaussian gradient noise. Freeze the coordinate w and let the momentum distribution settle to its steady state; this is stationary momentum. Let μ be the momentum coefficient and κ=B′B the batch ratio. The expected sign is A(B)=erf(wCNR·B(1+μ)2(1−μ)). Relative movement per processed sample is κα−1A(κB)A(B). We use reference B=1. Noise-dominated response grows approximately as B; signal-dominated response saturates. Preserving this local movement motivates square-root or linear scaling in the respective limits. The approximation freezes the parameter and assumes stationary momentum; it does not guarantee full-trajectory invariance. See Section 5 and Appendix C of the paper.

The sharp-subspace intervention.

Does a small set of directions account for the cost of increasing batch size in a language model? Building on the previous checkpoint-branching method12, we test different batch sizes in different matrix directions.

We compare a 128K-token control with updates scaled to 2M tokens. In other branches, we keep small-batch updates in the sharpest directions or in a random subspace. We select sharp directions using a local curvature estimate after Muon’s preconditioning. All other parameters keep their original AdamW updates.

Measuring the local batch-size penalty

Penalty
P=L−Lcontrol
Recovery
R=Pfull−PheldPfull
L
Validation loss at the end of a branch
Lcontrol
Loss of the matching 128K control
Pfull
Penalty with large-batch updates in every matrix direction
Pheld
Penalty with small-batch updates in the selected directions
R
Fraction of the fully scaled penalty removed

Branches start from the same checkpoint and use the same training data. Recovery measures a local loss penalty, not savings in total training cost.

Keeping small-batch updates in the sharpest 768 directions, about 0.001% of the matrix-parameter space, removes up to 59.5% of the local penalty. A random subspace of the same size changes it by at most 3%. The benefit largely disappears near the end of training.13

Takeaway: A tiny set of sharp directions can drive much of the local error from increasing batch size.

Directional branching.

~0.001%768 / ~85 million directions
1,000 · early12,000 · late

Branches from the same checkpoint.

Validation lossCheckpoint 5,000

Training step

A smaller penalty means the branch stayed closer to the small-batch control’s validation loss.

Protocol

These are the paper’s original held-out-loss curves: 1M-token evaluations every 32 base steps. The endpoints below use a separate 10.5M-token evaluation. Checkpoints 1K–11K continue for 1,024 base steps; the 12K checkpoint continues for 992 steps to the end of training. Top-128 was not run at 7K, 9K, or 11K.

The sharp directions have the largest eigenvalues of a Muon-preconditioned Gauss–Newton approximation to the loss Hessian, a local curvature proxy. Branches continue for 1,024 base steps, except the 12K checkpoint, which runs for 992 steps to the end of training. The penalty subtracts its small-batch control’s final validation loss.

The held subspace stays fixed throughout the continuation. Both Muon copies receive the same stream of base-batch gradients. Every 16 steps, the large-batch copy uses their mean gradient and 16 times their mean learning rate. Matrix weight decay and auxiliary AdamW updates stay on the base clock. This is a diagnostic intervention, not an ordinary large-batch training run.

Endpoints use 10.5M held-out tokens. Most branches are single runs, so these percentages are descriptive measurements, not significance estimates. The experiment selects directions by curvature and does not measure the full direction-dependent noise covariance. Its effect largely disappears near the end of training. We therefore cannot infer a 59.5% improvement in final loss, total training cost, or wall-clock speed.

Takeaways

Batch size can change which optimizer works best, even after retuning. We encourage researchers to design and test optimizers for the batch sizes they will actually be used with. Deriving optimizers and scaling rules that jointly consider gradient noise and the loss landscape is an important direction for future work.