Accelerating Open LLMs
through incentivized research.
Refinery accelerates open-source AI through a decentralized search for better training algorithms. We begin with the optimizer: the rule that turns gradients into learning.
The network brings the talent. The refinery tests their ideas. Better training algorithms improve the model.
Performance starts with
algorithms and data.
AdamW and Muon are milestones, not the endpoint. A wider search, accurate evaluation, and rewards for real improvements could move optimizer development substantially further.
Model capability has advanced dramatically. We believe there is still substantial headroom in the lower-level operations used to produce it. A better update rule can improve what a training run achieves with the compute it already has.
The optimizer is part of what makes that capability attainable. From stochastic gradient descent and momentum to adaptive methods such as Adam and AdamW, improvements to the update rule have helped shape modern training. In language-model experiments, optimizer choice can substantially change the loss reached within a fixed training budget—not just how quickly the same result arrives.
Published experiments: Kempner Institute’s LLM optimizer comparisons ↗
Useful algorithmic advances can come from academic research, open-source experiments, and independent contributors as well as frontier labs. Refinery makes this work an ongoing, incentivized search: miners propose improvements, a common evaluation measures them, and verified progress earns rewards.
The aim is stronger open LLMs: advances that compound across the training process and give open-source AI a stronger foundation. Better algorithms should become building blocks for everyone.
Read Refinery’s research thesis →Change the trajectory
of open models.
The remaining gap with closed systems is not only data. It is also the algorithms that decide how much a given budget of compute and data can produce.
Refinery exists to move the trajectory of open-source language models. Capability has advanced quickly, but we do not think the distance that remains is explained by data alone. We see substantial headroom in the algorithms, optimizers, kernels, the model architecture itself, and how inputs are processed. This includes components that the field has treated as settled for years.
Some of that headroom is already public. Looped transformers are a small, open example of an older idea still being pushed. FlashAttention and Muon are larger ones: methods that produced large gains in speed or training efficacy once they reached the open community. We believe there are more improvements of that kind, and that a wider, incentivized search can find them.
We start with the optimizer. From launch, for at least six weeks, miners are rewarded for update rules that beat AdamW and Muon on small models under a fixed budget. That is the first target, not the last.
We think this is a consequential moment. Whether comparable capability exists in the open will shape who can use it. Our aim is to use Bittensor to point a global pool of researchers at discoveries the open research community has not had in years, and to put those discoveries in everyone’s hands.
Open research.
Refinery’s ambition extends beyond a better optimizer: advance open-source AI, demonstrate what Bittensor’s global talent pool can produce, and bring more researchers into the network.
Advance research everyone can use.
Better training algorithms should empower the open-source world. By sharing methods, code and evaluation results, we aim to give researchers and builders stronger foundations for the next generation of open AI.
Make progress inspectable.
Use the network’s global talent pool to produce work others can check. Shared methods, code and evaluation results give the field a common record to build on.
Bring researchers into Bittensor.
Make those results an invitation. Give talented researchers a concrete route into mining: contribute an optimizer, test it against a shared benchmark, and earn rewards for confirmed progress while advancing open research.
The aim is a reinforcing cycle: Bittensor enables useful research; that work is published; and more researchers join as miners to push it further.
Explore contributing as a miner ↗A focused target.
A wide search space.
Our view is that optimizer ideas are more practical to compare in smaller training runs than changes to the model architecture itself or the kernels that execute it. Keeping the model and data fixed lets us ask a precise question: does this update rule train better within the same budget?
Explore beyond the defaults.
Miners can refine established methods or propose a different update rule. Human insight, automated experiments, and LLM research agents can all contribute to the search.
Standardize the environment.
Compare update rules under the same model, data, and compute budget. A shared evaluation environment makes results comparable, and a separate confirmation run checks that improvements hold up.
Direct a global pool of researchers.
Bittensor aligns a global pool of researchers around a shared target. Confirmed improvements earn rewards, while openly shared methods and evaluation records let everyone build on the work.
Global talent.
Shared progress.
Better training algorithms are already worth pursuing. Frontier model capabilities expand what researchers can explore through automated research. Refinery uses Bittensor to direct that global talent at specific problems, with common tests, rewards for progress, and discoveries shared openly.
Explore Refinery’s research approach →Compounded advances across
the set of algorithms
behind a training run.
Descent algorithms are our starting point. Pretraining data quality and reinforcement-learning environments are other components where a wider pool of contributors could make useful advances.
When we compare models from OpenAI with open-weight families such as Qwen and Kimi, we see the outcome of many interacting choices across training. Refinery’s ambition is to help close capability and efficiency gaps by improving those components, one measurable target at a time.
Explore the broader direction ↗Test the idea against results.
Emissions are available from day one, earned by verified progress. After six weeks, we will review the results and discuss whether a research pause would help the next stage. No pause is scheduled automatically.