Refinery is an incentivized gradient descent algorithm search. The goal is to use miners to improve the efficacy and efficiency of the lower-level operations behind LLM training.
We do not believe AdamW or Muon is the most efficient update rule that could exist for language-model training. The space of possible algorithms is much larger than the set in common use. Our thesis is that accurate measurement and sustained incentives can make exploring that space productive.
Some foundational methods remain central across generations of models. That persistence motivates us to revisit them. Research such as FlashAttention shows how rethinking a low-level operation can improve the efficiency of an existing architecture; work scaling Muon to LLM training shows that optimizer development remains an active source of progress.
Academic groups, open-source communities, independent researchers, and frontier labs all contribute to this work. Refinery asks what happens when a decentralized network has a clear objective, a credible test, and a direct reason to keep searching.
Research context: FlashAttention · Muon is Scalable for LLM Training
Modern training rests on accumulated algorithmic ideas. The original Transformer used Adam. Later comparisons show that Adam and other alternatives can reach substantially better language-model loss than standard SGD with momentum under the tested budgets. Optimizers are part of the machinery that makes effective training practical, not an interchangeable detail.
The original Transformer, §5.3 · LLM optimizer comparisons
This is a lineage of related ideas, not a sequence in which every method universally replaces its predecessor.
Gradient descent follows the local direction of decreasing loss. Stochastic gradient descent estimates that direction from sampled examples or minibatches, making updates possible without processing the entire dataset each time. Its simplicity provides a foundation, not a guarantee of efficient training in every setting.
Momentum accumulates information from previous gradients instead of treating each update independently. Classical momentum and Nesterov acceleration are related branches of this idea; both helped develop the toolkit for training deep networks.
AdaGrad uses accumulated squared gradients to adapt step sizes for individual parameters. RMSProp uses a moving average of squared gradients instead of an ever-growing sum. These methods address the fact that different parameters can need very different update scales.
Adam combines momentum-like averaging of gradients with an adaptive scale derived from their squared values, correcting the initial bias in both estimates. It draws on AdaGrad and RMSProp rather than discarding the gradient-descent foundation.
AdamW decouples weight decay from Adam’s adaptive gradient update. This addresses a specific mismatch: adding an L2 penalty is not equivalent to decoupled weight decay under Adam, although the two can be equivalent for standard SGD after rescaling.
These are mathematically motivated methods refined through experiments, not arbitrary tricks—and not proofs of the best possible optimizer for LLMs. AdamW itself illustrates how a small, well-motivated change to a familiar algorithm can become useful. Refinery’s conviction is that this process has further to go.
It would be too strong to say capable LLMs could not exist without Adam or AdamW. A 2025 study found carefully tuned SGD with momentum could perform similarly to Adam in small-batch language modeling. A 2026 preprint reports that stabilized, large-learning-rate SGD closes most of the validation-loss gap in its 1B-parameter experiment. Neither result establishes parity at every scale, but both challenge an absolute claim of necessity.
Batch size and the Adam–SGD gap · 2025 · Large effective learning rates · 2026 preprint
The important claim is about what a finite training budget can achieve. Better optimization can make lower loss attainable within that budget. Lower loss is not a complete measure of capability, but improving the training process is a concrete route toward stronger Open Intelligence—and a reason to keep searching beyond today’s defaults.
Changing the function approximator—the model architecture—or its compute kernels introduces a different set of evaluation challenges. We believe optimizer search offers a more practical first target for smaller training runs: hold the model, data, and compute budget fixed, then compare update rules.
The production task trains a 360M-parameter language model for up to twenty hours on one B200. The learning-rate schedule belongs to the submission. Held-out loss measures the result, so an optimizer must justify its own overhead within the same compute budget.
This is a bounded experiment with an observable outcome. It gives miners room to explore and the validator a common basis for deciding what improved.
Read the evaluation contract →Our working hypothesis is that training experiments costing roughly $100–$1,000 in compute can reveal relative optimizer performance that transfers to runs with 10–100× larger training budgets, at similar model sizes. This refers to a larger run budget, not a claim of transfer to models 10–100× larger.
That transfer must be tested. Optimizer rankings may change with training duration, data, hyperparameters, and hardware. We believe the subnet’s benchmark is the best way to measure a gradient descent algorithm’s performance within the limited compute and time available to each run.
We believe Bittensor can attract the talent to make meaningful algorithmic improvements. Miners can bring their own research judgment, experiment infrastructure, and approaches to the problem.
Before this subnet design, the team ran exploratory experiments using LLM agents to search for improvements over standard optimizers on small training runs. The results were encouraging enough to pursue the direction with miners.
Those local findings are preliminary. They are motivation for the experiment, and we would not place too much weight on them before seeing what miners produce at scale. The stronger evidence will be independently confirmed improvements, published records, and follow-up evaluations.
As research agents improve, the set of viable tasks can change. Optimizer search is where we are choosing to test that possibility now.
The broader goal is to improve the components of model training through decentralized contribution. Better descent algorithms, better pretraining data, and better reinforcement-learning environments can reinforce one another. Together with architecture and compute, these choices help shape the differences people observe between model families.
Discover more effective update rules under a fixed evaluation budget.
Explore quality, filtering, and mixture choices that make training examples more useful.
Explore tasks and feedback that help models learn more useful behavior through practice.
Follow the optimizer search through its submissions, confirmed frontier changes, and published evaluation records.