Every major LLM on the market today, ChatGPT, Claude, Gemini, all of them, runs on the same nine-year-old idea: the transformer, introduced in a 2017 Google paper with the now-famous title “Attention Is All You Need.”
It has been, without exaggeration, one of the most consequential ideas in computer science. It is also, per a new MIT Technology Review deep dive, starting to buckle under its own success. A wave of well-funded startups now think they’ve found what comes next, and none of them agree on what that is.
Major Highlights
- The bottleneck: transformers process text by comparing every word against every other word, a mechanism called dense attention. A 10,000-word document can require roughly 50 million such comparisons, which is a major reason OpenAI expects to spend $50 billion on compute this year alone.
- Subquadratic (Miami) claims the first sparse attention mechanism, which skips comparing every word to every other word, that rivals mainstream LLMs on tasks like search and coding. The claim is contested by other researchers.
- Manifest AI (San Francisco) built power retention, which keeps only a rolling summary of relevant context instead of the whole conversation history. It has released PowerCoder and Brumby, claiming performance on par with versions of Alibaba’s Qwen.
- Liquid AI, an MIT spinout, blends transformers (20%) with liquid neural networks inspired by worm brains (80%). Its models run Mercedes vehicle systems and fit on a $50 Raspberry Pi, and have been downloaded roughly 34 million times.
- Inception (Palo Alto) applies diffusion, the tech behind image generators, to text, producing whole blocks at once instead of word by word. Its Mercury 2 model claims GPT-4-level performance at 10x the speed.
- Pathway (Palo Alto) built Dragon Hatchling, which replaces attention with a mathematical state space and beat 97% of 250,000 hard sudoku puzzles that stumped several leading LLMs outright, arguing language itself is the wrong tool for some kinds of reasoning.
KINI BIG DEAL?
Strip away the architecture jargon and this is fundamentally a story about cost, not cleverness. OpenAI spending $50 billion on compute this year is the number that explains why five separate startups think there’s room to out-engineer the biggest labs in the world on efficiency rather than raw scale. Nobody is trying to build a smarter ChatGPT here; they’re trying to build a cheaper one, or a faster one, or one that fits somewhere a transformer physically can’t.
That framing matters more in Lagos than in Palo Alto. Most of the continent isn’t going to out-compute anyone, and most African businesses, schools, and clinics don’t need to. What actually reaches them is the efficiency story: Liquid AI’s models running on a $50 Raspberry Pi, or a diffusion model doing GPT-4-level work at a tenth of the cost, is a far more useful data point for a Nigerian edtech startup or a rural clinic’s diagnostic tool than another headline about parameter counts. Cheaper, smaller, and offline-capable is the version of “the next big thing in LLMs” that actually ships on the continent, not the version that needs a $50 billion compute budget behind it.
The honest caveat, and it’s a real one: none of these five have actually dethroned the transformer yet. Subquadratic’s claims are disputed by researchers in the same MIT piece, Manifest AI is benchmarking against open models rather than frontier ones, and every mainstream tool anyone actually uses today — ChatGPT, Claude, Gemini — is still built the old way. Zero agreement isn’t just a headline; it’s the accurate state of the field right now. Worth tracking, not worth betting the syllabus on yet. Na who go first prove say dem sabi pass transformer go carry the market.
