An Internal Experiment in Model Orchestration
Where orchestration beats a single frontier model, and where it does not.
On the 12th of June 2026, Fable 5 was pulled from the shelf, stripped away from its users. If you were an agentic user, you know the feeling: one week you had a breakthrough model doing your hardest work, and the next day it was just gone…1
Overnight, it set off a whole debate about AI access, security and sovereignty. For the first time, a country had reached in and cut off access to a frontier model. Never before had access to AI been switched off like that, and in the scramble for a replacement, a new class of AIs stepped into the spotlight: orchestrators and mix-of-agents models, each claiming to deliver Fable-like capability.
The idea is simple: instead of a smarter model, they combine a handful of models ("lesser" or frontier) to get a better result.
The two best-known examples of this are OpenRouter's Fusion and Sakana's Fugu Ultra. Both of them, and a growing list of competitors, sell the same two promises.
- Similar or better performance to frontier models
- At lower costs
That's exactly what we want to find out today, and beyond that, we have two more questions we want to answer.
- Is a mixture of models actually better than just calling a single frontier?
- And if not, is there any real edge in how we combine them?
Before we start, here's a short introduction to what Fusion and Fugu Ultra are.
Fusion was released on June 12 2026, the same day Fable was pulled. From the outside, it works like any other model. You send a prompt, you get an answer. Inside, it sends that prompt to a panel of frontier models, has another model judge their answers, then a final model writes the answer you actually see.
Fugu Ultra, on the other hand, runs on the same idea but goes a step further. A small coordinator model decides which models to bring in and how to split the work between them, handing out roles like thinker, worker, and verifier over a few turns. Sakana trained that coordinator with reinforcement learning and wrapped the whole thing behind a single model name.
Ok, now that that is out of the way, let's take a look at how we can test this.
The participants
A single frontier model: Claude Opus 4.8. Every mixture needs a ceiling to beat. Opus was the strongest model available for the whole test window, and it is the model the orchestrators themselves reach for when they want quality. Can a mix of agents beat the single best model in the pool? It is also the most expensive per token, which is what makes the comparison worth running.
A single cheap model: Gemini 3.5 Flash. The floor of the experiment. Flash-class models are what orchestrators actually put in their panels, ours included, so it doubles as a control. There is no point in having a panel if it scores below its cheapest component. Per token, it costs a small fraction of Opus.
Fusion. OpenRouter's commercial mixture of agents. You call it like any other model, and behind the API, it fans your prompt out to a panel, compares the answers, and writes one reply. It is the productised version of the architecture we are testing, sold on the promise of frontier quality at a lower price.
Fugu Ultra. Sakana's learned orchestrator. Where Fusion mixes a fixed panel, Fugu trains a controller to decide how to spend compute on each query. Its headline coding score is 93.2. That beats Opus; can we replicate the result in our tests?
bundle.2 Our own mixture of agents: a panel of Gemini 3.5 Flash, DeepSeek V4 Flash and Opus 4.8, a Gemini judge that compares the panel's answers, and an Opus synthesiser that writes the final reply. We built it so we could swap any part and meter every call. The commercial orchestrators are black boxes, so bundle is the result of reverse engineering Fusion and Fugu Ultra.
The tests
Humanity's Last Exam, text only. Hard multiple-choice reasoning, 40 questions from the hardest pool. Text-only, so vision quality does not pollute a reasoning comparison. Graded by exact letter match with the official harness. This was chosen because it's one of the few benchmarks that is not saturated and still works fairly well between models. It's worth noting that we only ran 40 hard questions instead of the whole set.
LiveCodeBench. 50 coding problems, hardest tier only, restricted to problems published after the models' training cutoffs. Grading is mechanical: the generated program passes the official hidden test suite in a sandbox, or it fails. Similarly, we only ran 50 problems here.
Shared rules for every arm. Cost is the real billed amount from the provider, never a price-sheet estimate. Latency is measured one query at a time, so timings are clean. Token budgets were set high enough that no model was cut off mid-answer.
Known limitations. We publish these so you can weigh the results yourself:
- Every arm ran once. This was a budget call. At 40 and 50 questions, a few points of difference can be luck — read the double-digit gaps as the signal and the small ones as noise.
- Subsets, not full suites. HLE: 40 questions from the official gated set — hard-filtered, then a random draw. Per the dataset's terms we don't republish questions, but the exact selection procedure and question IDs are documented. LiveCodeBench: the 50 newest problems, all dated after every tested model's training cutoff, so nothing was answerable from memory.
- Test harnesses: generation ran through our own checkpointed rigs, one query at a time, identical for every arm. Code was graded by the official LiveCodeBench evaluator — real execution, official pass@1. HLE multiple-choice was graded by exact letter match in our own harness, not HLE's official judge-model pipeline. This is also why we flag Fugu Ultra's non-finish as possibly a harness issue rather than a verdict.
- Single-turn, English, benchmark-shaped prompts. No multi-turn conversations, no production traffic. The results say what they measure and nothing more.
Everything you need to re-run any arm — harness, configs, question IDs, run commands — is in the repo.2
The results
TLDR: Orchestration loses on reasoning and wins on coding.
On reasoning, the single frontier model beats every mixture by double digits while costing the least and answering quickly. The two orchestrators that finished land close to the level of the cheap model they are built from.
On coding, the order flips: both mixtures clear the best single model (by eight points).
For Fugu Ultra, unfortunately, we couldn't get it to answer. It either kept timing out or did not produce an answer reliably enough. To be clear, this 100% could be a harness issue, but given that the other models answered with no issues, we found it not worthwhile to optimise for Fugu Ultra.
This experiment yielded a whole bunch of interesting results. How does the same architecture win one test and lose the other? What are the trade-offs in time and cost? These are all summarised in our findings.
Findings
Finding 1: Is orchestration any better? Yes and no.
It depends entirely on the task.
On coding, orchestration wins, and the results are clear. We believe that these models perform better in a coding environment, and code output can be standardised and formatted relatively easily. This makes it easier for the synthesiser to digest and make something real out of it.
coding: bundle 82%, fusion 78%
both above every single model
reasoning: every orchestrator near 50%
below opus at 62.5%
A smarter mix should theoretically win. Routing to it is the hard part.
Different models genuinely know different things. The cheap model is right on 13% of the questions Opus gets wrong. But we could not find a way to route this reliably. If we zoom into each individual question, the "best" model flips randomly with no apparent category or topic pattern.
some model was right on 82.5% of questions
(not a reachable ceiling: models are stochastic,
and lucky draws count toward that number)
opus alone: 62.5%
So there is a theoretical benefit in routing, but capturing it requires a strong predictor or classifier, and we have not built one.
Finding 2: Token cost is not the total cost (efficiency matters)
Opus has by far the highest cost per token, but surprisingly, it wasn't always the most costly per question. There are two hidden parts to separate here: reasoning tokens, the "thought process" that a model goes through, and the answer itself.
one HLE question
opus: 3 tokens $0.0007
cheap model: ~2,000 reasoning tokens $0.018
same answer. the premium model was 24x cheaper
Where a "cheaper" model would take thousands of tokens to come up with an answer, Opus simply answers it. Token efficiency directly impacts costs.
Finding 1 still applies: on coding, it becomes a lot more expensive, because all participants go through a lot of reasoning when writing code, and Opus no longer benefits from brevity.
Finding 3: The bottleneck is the synthesiser, not the panel
Let's compare the orchestrators. Bundle and Fusion are fundamentally the same structure; the only real difference is which models sit in the pool.
Every combination we tried pointed to the same conclusion: whoever writes the final answer matters more than whoever sits on the panel.
The sharpest proof: our strongest bundle had Opus
in the pool and used Opus as the synthesiser,
and it still scored only 50% on reasoning,
below a plain Opus call at 62.5%.
Increasing the size of the panel didn't help either. Cheap models tend to make the same mistakes and will agree on the same wrong answer. It's no surprise that the sum of models is limited by the strongest one, but if you have only one place to put it, choose the synthesiser.
Finding 4: Benchmarking is genuinely hard
While conducting this experiment, we found that most public benchmarks are dead for today's models. They are either saturated, where everyone scores near 100%, or contaminated, where the models have memorised the answers. We went through several before finding two that still separated the models.
Furthermore, how much do these results actually reflect the experience of an average user? Nobody asks a model 50 hard competition problems for fun, and the questions that fill HLE will rarely come up in anyone's day. Benchmarks are academic by nature, which is fine for an experiment like this one, but we need better benchmarks.
Take this, for example: a simple prompt for a water gun game, run side by side, shows how differently the models approach the same problem. Again, we need better benchmarks!
Future work
One thing that surprised us was that bundle could, in some cases, outperform the commercial orchestrators already out there. Sadly, most of them are closed boxes with nothing for the user to customise. Now imagine a router that anyone can tune.
Three directions that we want to try to take this further:
- Tiering and classification. Do we really need a panel for all problems? Can we reliably detect the "difficulty" of a prompt before escalating to a panel? Is that faster or cheaper?
- Compressed prompting for token savings. A lot of an orchestrator's cost is boilerplate: system prompts, instructions, and communications between models. Writing that in a stripped-down, telegraphic language instead of full sentences cuts the token count directly. In short:
- A verification loop for code. Running code is nearly free in terms of tokens, so can we implement a loop similar to Fugu Ultra's to improve the quality of code? This is the step that turns bundle from a mixture of agents into a harness.
Before we close, finding 4 also said we need better benchmarks, and we stand by it. But benchmarking is a rabbit hole of its own and a little out of scope for this article. We are also working on Calathea to bring better benchmarks to the average AI user. Side-by-side tests on real prompts are the most promising direction we have seen for closing the gap between benchmark scores and actual use. We're excited about the work we are doing there, and we'll share more about it another time.
We've learnt a whole lot from this experiment, and hopefully you did too. There are still many unsolved problems in this landscape of AI and agents. If you have any ideas or comments, reach out at [email protected]. We would love to hear them!
1 Fable 5 was re-released during the writing of this article. Everything this study measured happened in the window it was gone.
2 bundle is open source at github.com/nbitslabs/bundle-oss. The study's final benchmark ran on 1 July 2026.