Refinery is an incentivized gradient descent algorithm search. The goal is to use miners to improve the efficacy and efficiency of the lower-level operations behind LLM training.
Better training algorithms are already worth pursuing. Frontier models expand the capacity to pursue them: researchers can use coding and reasoning agents to propose changes, run evaluations, and iterate on the results. Work such as AlphaEvolve demonstrates that automated algorithm search can produce useful improvements, including to model training.
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Refinery brings that opportunity to Bittensor’s global pool of researchers. Clear targets, standardized environments, and rewards for confirmed progress give contributors a common direction while leaving them free to choose their methods. Human expertise and automated research can work together to explore more of the space.
The goal is to make this research open. Optimizer code, methods, and evaluation results should become shared building blocks, so improvements discovered through the network can advance open LLMs and give the next researcher a stronger starting point.
We do not believe AdamW or Muon is the most efficient update rule that could exist for language-model training. The space of possible algorithms is much larger than the set in common use. Our thesis is that accurate measurement and sustained incentives can make exploring that space productive.
Some foundational methods remain central across generations of models. That persistence motivates us to revisit them. Research such as FlashAttention shows how rethinking a low-level operation can improve the efficiency of an existing architecture; work scaling Muon to LLM training shows that optimizer development remains an active source of progress.
Recent interest in looped transformers points to another part of this opportunity: the model architecture itself is still evolving. Reusing transformer layers for successive reasoning steps changes how the model computes. Even the function approximator, along with the algorithms used to train it, remains an active research target.
Academic groups, open-source communities, independent researchers, and frontier labs all contribute to this work. Refinery asks what happens when a decentralized network has a clear objective, a credible test, and a direct reason to keep searching.
Research context: FlashAttention · Muon is Scalable for LLM Training · Reasoning with Latent Thoughts: On the Power of Looped Transformers
Modern training rests on accumulated algorithmic ideas. The original Transformer used Adam. Later comparisons show that Adam and other alternatives can reach substantially better language-model loss than standard SGD with momentum under the tested budgets. Optimizers are part of the machinery that makes effective training practical, not an interchangeable detail.
The original Transformer, §5.3 · LLM optimizer comparisons
This is a lineage of related ideas, not a sequence in which every method universally replaces its predecessor.
Gradient descent follows the local direction of decreasing loss. Stochastic gradient descent estimates that direction from sampled examples or minibatches, making updates possible without processing the entire dataset each time. Its simplicity provides a foundation, not a guarantee of efficient training in every setting.
Momentum accumulates information from previous gradients instead of treating each update independently. Classical momentum and Nesterov acceleration are related branches of this idea; both helped develop the toolkit for training deep networks.
AdaGrad uses accumulated squared gradients to adapt step sizes for individual parameters. RMSProp uses a moving average of squared gradients instead of an ever-growing sum. These methods address the fact that different parameters can need very different update scales.
Adam combines momentum-like averaging of gradients with an adaptive scale derived from their squared values, correcting the initial bias in both estimates. It draws on AdaGrad and RMSProp rather than discarding the gradient-descent foundation.
AdamW decouples weight decay from Adam’s adaptive gradient update. This addresses a specific mismatch: adding an L2 penalty is not equivalent to decoupled weight decay under Adam, although the two can be equivalent for standard SGD after rescaling.
These are mathematically motivated methods refined through experiments, not arbitrary tricks—and not proofs of the best possible optimizer for LLMs. AdamW itself illustrates how a small, well-motivated change to a familiar algorithm can become useful. Refinery’s conviction is that this process has further to go.
Optimizer search offers a large function space to explore: update rules can change how gradients, parameter values, and accumulated state interact over time. We believe it is a practical first target for smaller training runs. Holding the model, data, and compute budget fixed gives us a common basis for comparing ideas before broadening the search to architectures and kernels.
The production task trains a 360M-parameter language model for up to twenty hours on one B200. The learning-rate schedule belongs to the submission. Held-out loss measures the result, so an optimizer must justify its own overhead within the same compute budget.
We believe there is room for optimizer gains well beyond ordinary run-to-run noise. The search targets those meaningful improvements across a much larger space of possible functions than today’s standard methods cover. Candidates must clear an adaptive improvement threshold, with a floor of 0.03 nats, in both the initial evaluation and a separate confirmation run.
Read the evaluation contract →Our working hypothesis is that training experiments costing roughly $100–$1,000 in compute can reveal relative optimizer performance that transfers to runs with 10–100× larger training budgets, at similar model sizes. This refers to a larger run budget, not a claim of transfer to models 10–100× larger.
That transfer must be tested. Optimizer rankings may change with training duration, data, hyperparameters, and hardware. We believe the subnet’s benchmark is the best way to measure a gradient descent algorithm’s performance within the limited compute and time available to each run.
We believe Bittensor can attract the talent to make meaningful algorithmic improvements. Miners can bring their own research judgment, experiment infrastructure, and approaches to the problem.
Before this subnet design, the team ran exploratory experiments using LLM agents to search for improvements over standard optimizers on small training runs. The results were encouraging enough to pursue the direction with miners.
Those local findings are preliminary. They are motivation for the experiment, and we would not place too much weight on them before seeing what miners produce at scale. The stronger evidence will be independently confirmed improvements, published records, and follow-up evaluations.
As research agents improve, the set of viable tasks can change. Optimizer search is where we are choosing to test that possibility now.
The broader goal is to improve the components of model training through decentralized contribution. Better descent algorithms, better pretraining data, and better reinforcement-learning environments can reinforce one another. Together with architecture and compute, these choices help shape the differences people observe between model families.
Discover more effective update rules under a fixed evaluation budget.
Explore quality, filtering, and mixture choices that make training examples more useful.
Explore tasks and feedback that help models learn more useful behavior through practice.
Follow the optimizer search through its submissions, confirmed frontier changes, and published evaluation records.