Why can a 27B model beat larger frontier systems?
Model size is only one part of an AI system. Inherent says its Faraday agent, built on Qwen 3.6 with 27 billion parameters, outperformed Claude Opus 4.8 and GPT-5.5 at independently reproducing findings from published scientific papers. That is striking because parameter count is commonly used as a rough proxy for a model’s scale and training cost.
The less obvious point is that the comparison was not simply one language model answering one question. Faraday is an agent designed for a particular job: reproducing research. It can use GPT-5.5 Codex as its coding tool, rather than relying on an in-house coding model. This means the result reflects the performance of a complete workflow, including the underlying model and the tools it uses, not just the number of parameters in its base model.
That distinction matters in practical work. A larger general-purpose model may have more capacity, but capacity alone does not guarantee that it will organize a research task well, write the needed code, run the right experiments, or check whether the result matches the paper. A smaller system with a focused process can perform better on a narrow evaluation.
The claim also needs to be read precisely. Inherent reported that Faraday beat these frontier systems on replicating published research findings. It does not show that a 27B model is broadly stronger than Claude or GPT-5.5 across all tasks.
The takeaway is simple: bigger models can provide more raw capability, but agents win through task design. For scientific replication, the workflow around the model may matter as much as the model itself.
What exactly did Faraday replicate?
Faraday was tested on whether it could reproduce research described in published scientific papers, not merely summarize those papers or answer questions about them. Inherent created Replica, a benchmark containing 310 tasks drawn from 100 scientific papers. The benchmark was designed to measure how well an AI agent could carry out research replication over extended tasks.
That distinction matters. A language model can often explain a paper without proving that it can recreate the work behind it. Replication requires a chain of actions, including understanding the research goal, working through the relevant material, and producing an outcome that can be judged against the original study. The available research does not list all 310 task instructions, so it would be wrong to claim that every task involved the same kind of experiment or code.
Replica separates familiar and unfamiliar tests. Faraday reportedly succeeded on 73% of in-distribution tasks, which resemble the material used during development, and 60% of held-out tasks, which provide a test of performance on research problems it did not train directly against. Those scores are the basis for Inherent’s claim that a 27-billion-parameter Faraday system outperformed Claude Opus 4.8 and GPT-5.5 on the benchmark.
The result is therefore narrower, and more useful, than a claim that Faraday is generally a better scientist. It shows strong performance on a defined set of paper-replication tasks. Replica’s rubric-based automated judge scores the agent’s long-running work, making it possible to compare systems on more than a single answer.
The takeaway is simple: Faraday replicated benchmarked research tasks drawn from real papers. It did not replicate every kind of scientific work, and the reported result depends on Inherent’s own benchmark and scoring system.
How Replica turns 100 papers into 310 research tasks
Published papers often describe a result without showing every decision that produced it. The missing details may include how an experiment was configured, which data choices mattered, or how the result was checked. Replica tests whether an AI agent can recover that hidden experimental work instead of merely repeating a paper’s conclusion.
The benchmark contains 310 tasks drawn from 100 machine-learning and AI-for-science papers. Each task is built around reproducing a published result, and the agent is not given the answer in advance. That condition matters. A system can sound convincing while quietly relying on the expected outcome. Replica instead asks it to do the work needed to reach the result.
The benchmark’s framework, described in an arXiv paper, also includes an auto-generated rubric judge. This judge provides a consistent way to assess whether an attempted replication meets the task’s requirements. The result is a test of practical research ability, not just language fluency or paper comprehension.
Replica is also designed around a larger idea: replication is a stepping stone toward innovation. Before an agent can improve an experiment, it must first reconstruct what the original researchers actually did, including details that may not appear clearly in the paper.
That makes the benchmark useful for a specific reason. It measures whether an agent can turn published knowledge into working experiments. On Inherent’s benchmark, Faraday, a 27-billion-parameter agent built on Qwen, outscored Claude Opus 4.8 and GPT-5.5 on these held-out replication tasks. The key lesson is simple: strong research agents must recover the work behind results, not just describe the results themselves.
Why the agent workflow matters more than the base model
A model’s size is easy to measure. Its ability to complete useful research is harder, and that difference matters. Inherent’s Faraday is a 27-billion-parameter agent built on the open-weight Qwen 3.6 model. It reportedly outperformed Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5 at independently reproducing findings from published scientific papers.
The surprising part is not simply that a smaller system beat larger frontier models. It is that the test focused on a complete research task rather than a single answer. Reproducing a paper requires more than recalling information. The system must work through the paper’s claims and reach the same findings independently. The result suggests that performance can depend heavily on how a model is organized around a task, not only on how many parameters it has.
That distinction changes how teams may evaluate AI systems. A larger base model can be valuable, but size alone does not guarantee better research outcomes. If an agent is designed to carry out a defined investigation, its workflow may matter as much as its underlying model. This also challenges the assumption that buying more compute, or choosing the largest available model, is always the best path.
There is a limit to the lesson. Faraday’s reported advantage applies to a specific benchmark: reproducing published scientific research. It does not prove that a 27-billion-parameter model is broadly stronger than the systems it beat.
The practical takeaway is simple: evaluate the whole research process, not just the model underneath it. The winning system may be the one that completes the job most reliably, even when its base model is smaller.
What Faraday uses GPT-5.5 Codex to do
GPT-5.5 is not described as part of Faraday’s architecture. In the reported test, OpenAI’s GPT-5.5 served as one of the systems competing against Faraday, alongside Anthropic’s Claude Opus 4.8.
The setup was deliberately simple. Each system received the same published scientific papers and was asked to reproduce the findings independently. None was told the correct result in advance. This matters because the task is not just summarizing a paper or writing code that looks plausible. The system has to decide which experiments to run, carry them out, and reach the paper’s reported conclusion.
That makes GPT-5.5 a useful benchmark for Faraday. GPT-5.5 is described as a much larger, frontier-scale model, while Faraday is a 27-billion-parameter agent built by London-based Inherent on the open-weight Qwen 3.6 model. The comparison tests whether a smaller system, trained for a specific kind of work, can reproduce research as reliably as systems backed by far more compute.
The surprising point is that model size did not settle the contest. Inherent says Faraday outperformed both Claude Opus 4.8 and GPT-5.5 on the replication task. Its claimed advantage comes from reinforcement learning that rewards successful outcomes, rather than teaching a fixed list of correct steps. Inherent calls the resulting ability “research taste”: knowing which experiments are worth running and how to design them well.
The practical takeaway is clear: GPT-5.5 is the opponent in this evaluation, not the engine inside Faraday. The test is less about who can generate the most polished answer and more about who can make better experimental decisions.
How long-horizon reinforcement learning improves scientific work
Scientific research rarely succeeds in one step. A researcher must choose an experiment, run it, inspect the result, revise the plan, and decide what to try next. Reproducing a paper follows the same pattern. The agent cannot simply state the published conclusion. It must reach the result without being given the answer in advance.
That makes replication a useful test for long-horizon reinforcement learning. The important skill is not only producing correct code or explaining a paper. It is staying oriented across a chain of decisions, where an early mistake can waste later work. Faraday’s task tests whether an agent can turn a paper into a working investigation, then use the evidence from that investigation to guide its next move.
The surprising part is that success depends on judgment as much as execution. Inherent wanted Faraday to show “research taste”: an instinct for which experiments are worth running and how to design them well. That is close to how human scientists learn. Edward Hughes said many PhD students begin by reproducing existing research before trying to produce new knowledge.
This also explains why model size alone did not settle the result. Faraday uses Qwen 3.6, a 27 billion parameter model, yet Inherent says it outperformed much larger Claude Opus 4.8 and GPT-5.5 systems on independently reproducing published findings. The system’s advantage may come from how it works through the research process, not just from how much information its underlying model contains.
The practical takeaway is simple: scientific agents need training that rewards sustained investigation and good experimental choices. A capable model can write code, but a useful research teammate must know what to test next.
Why held-out results matter more than familiar benchmarks
A benchmark can show that an AI system performs well on tasks it has already seen, or on questions with familiar patterns. Faraday was tested differently. Inherent says the agent independently reproduced findings from published scientific papers without receiving the answers in advance. That makes the result more relevant to research work, where the system must decide what to investigate and then determine whether its results support an existing claim.
The distinction matters because Faraday was not asked to produce new scientific knowledge. It was given a rehearsal for the larger goal: an agent that can contribute to discovery across scientific fields. Reproducing known findings is a narrower task, but withholding the answers tests whether the agent can carry out the work rather than repeat a stored solution.
In Inherent’s evaluation, Faraday reportedly outperformed Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5. Those results are company-reported, so they should be read as evidence from Inherent’s own test, not as a universal ranking of scientific ability. Still, the setup points to a useful standard. A research agent should be judged by what it can discover when the expected result is hidden, not only by how fluently it responds to familiar prompts.
This also explains why Inherent wants Faraday to behave more like a curious teammate. The desired agent does not simply tell users what they want to hear. It returns with experiments it chose to run and results that can be examined. The takeaway is simple: held-out replication tests whether an agent can do the work, not merely recognize the answer.
Can an automated judge reliably score research replication?
A replication result is only as useful as the test that decides whether it succeeded. Faraday was asked to recover findings already reported in published papers, without receiving the answers in advance. That creates a concrete target, but it does not automatically make the evaluation reliable.
The difficult part is deciding what counts as a match. A research agent may reproduce a headline result while using different data handling, experimental settings, or statistical choices. It may also produce a result that is directionally correct but not identical to the original paper. The available information does not explain how Inherent judged these cases, whether the score came from an automated system, human researchers, or both.
That missing detail matters. Inherent says Faraday beat Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5, but it has not provided scores, the papers used, or the evaluation method. Without those details, the comparison cannot show how often each system replicated results, how difficult the tasks were, or whether the judge rewarded genuine experimental work rather than a plausible report.
There is another complication: Faraday delegates coding to OpenAI’s GPT-5.5 Codex. The result therefore evaluates the full system, including model selection, reinforcement learning, experiment planning, and delegation. It does not cleanly measure the ability of the 27 billion parameter Qwen 3.6 core model alone.
The takeaway is simple: replication needs more than a pass or fail label. A trustworthy benchmark should reveal the papers, scoring rules, success rate, and human checks. Until then, Faraday’s result is an interesting company claim, not yet a fully inspectable measure of research ability.
What this changes for AI research teams and operating costs
AI research teams may not need to scale model size first. Faraday, Inherent’s 27B-parameter agent, is reported to reproduce findings from published scientific papers while running on Qwen 3.6 and delegating coding work to OpenAI’s GPT-5.5 Codex. The unusual part is that the smaller model is paired with long-horizon reinforcement learning and specialized tool use, rather than being expected to solve the entire task in one response.
That changes the operating question. A team evaluating research agents should ask not only, “How large is the model?” but also, “Which parts of the workflow does each model handle?” Faraday’s design uses Qwen for the agent and GPT-5.5 Codex for coding. This may reduce the need for a single large model to manage planning, reasoning, and implementation at once. However, the available claims do not provide enough information to calculate savings, compare inference costs, or determine how much coding delegation contributes to performance.
The practical impact is a shift toward system-level budgeting. Teams may spend less effort searching for one model that does everything, and more effort assigning different models to distinct jobs. That could affect infrastructure choices, model contracts, evaluation processes, and staffing. Inherent is a London team of about a dozen people, funded by a $50 million seed round, with plans to grow to 20 to 25 employees by year-end. Those figures show a lean operating model, but they do not prove that Faraday is cheaper to run than larger alternatives.
The clearest takeaway is restraint: smaller models and careful orchestration may improve efficiency, but independent cost and performance data are still needed.
Which claims still need independent verification?
Faraday’s reported results are impressive, but they are still claims from one research team. The central finding is that a 27B-parameter agent outperformed Claude Opus 4.8 and GPT-5.5 on paper replication tasks. Independent researchers would need to reproduce the same result using the same task suite, scoring method, model settings, and evaluation process.
The comparisons also deserve careful checking. Claude was run through the Claude Code harness, while GPT-5.5 was run through the Codex harness, with thinking effort set to extra high. Those details matter because an agent’s tools, time, and execution environment can affect its performance. A fair replication would need to confirm that the harnesses and settings were comparable, rather than treating the model names alone as the experiment.
Faraday reportedly performed better across every paper category, with especially large advantages in meta-learning, structural biology, and materials science. It also reportedly struggled less with recent research and performed well on held-out AI-for-science domains. These findings need testing on new papers and new domains selected by researchers who did not build the benchmark. Otherwise, it remains difficult to separate broad scientific ability from familiarity with the task design.
Most importantly, replication is not the same as making a new discovery. The research presents Faraday as a step toward AI scientists capable of innovation, but the reported evidence concerns reproducing research. Independent work must establish whether those skills transfer to forming useful hypotheses, designing experiments, and producing reliable new results.
The takeaway is simple: the benchmark result may be strong, but its reach is not settled. Reproduction by outside teams is needed before Faraday’s advantage can be treated as a general fact about AI research agents.