Dust: Pretraining Transformers Without Backpropagation
What happened
Researchers introduced Dust, a zeroth-order optimization method that trains transformer language models without backpropagation by perturbing activations at each token and evaluating them in parallel. Dust approximates backprop gradients closely when using large populations, sometimes even outperforming backprop, and is far more efficient than weight-space evolution strategies like EGGROLL. The method scales better with larger models, with a 243 million‑parameter model beating a much smaller one across population sizes. Experiments show Dust’s gradient estimates stay well aligned with backprop up to one billion tokens. The work suggests that, given enough compute, brute‑force search methods could rival or surpass traditional gradient‑based training.
Why it matters
If such methods become practical, they could lower the reliance on differentiable architectures, potentially enabling new model designs and reducing the massive data needs of current AI systems.
Reality check
An automated, sceptical read of the sourceNotable. The approach demonstrates competitive performance and efficiency gains over existing ES methods, but it remains a research prototype rather than a widely adopted training technique.
Is this a big deal?
Get stories like this every morning
The day's AI news in plain English, in your inbox at 8:30 IST. Free.
More AI news
AI unlocks Gödel's notebooks, said to contain hundreds of proved theoremsIndustry news · 13 Nov 2026Gemini 4 Hydrogen and OrchestratorIndustry news · 12 Nov 2026Gemini 4 HydrogenIndustry news · 12 Nov 2026I'm making LLMs work on small and old devicesIndustry news · 12 Nov 2026United Airlines bans AI agents from booking flights on behalf of travelersIndustry news · 11 Nov 2026This page was collected from qlabs.sh and summarised automatically by AIPulsee's pipeline. See how it runs.