Why reproducing a paper is harder than explaining it
Explaining a research paper can stop at a clear summary. Reproducing it cannot. An agent must turn the paper into working code, identify the right data and settings, run experiments, interpret the results, and handle problems that the authors may not spell out. The goal is not to sound convincing. It is to obtain the result.
That difference matters because published research often compresses a long process into a few pages. A paper may describe a method, but reproducing it requires many connected decisions. Which implementation details matter? Which parts of the experiment are essential? What should happen when a script fails or a result does not match? A capable agent must keep working across these steps rather than produce one polished response.
Inherent designed Replica around this difficulty. The benchmark contains 310 tasks drawn from 100 machine-learning and AI-for-science papers. Its tasks test whether an agent can reproduce results without receiving the answer in advance. A rubric-based, automatically generated judge scores the work, which gives Inherent a way to train and evaluate agents on long sequences of research actions rather than on short answers.
That also explains why model size alone is a poor guide. Faraday uses a 27-billion-parameter Qwen base, yet Inherent reports that it outperformed Claude Opus 4.8 and GPT-5.5 on 73% of in-distribution tasks and 60% of held-out tasks. Faraday can also call GPT-5.5 Codex for coding subtasks.
The practical lesson is simple: research automation is measured by finished, checkable results, not by how well an agent describes the path to them.
What Faraday actually does from paper to experiment
Most AI systems are judged by how well they answer questions about research. Faraday is tested on a harder task: whether it can recover enough of a paper’s experimental work to reproduce its findings.
The process begins with a published scientific paper. Faraday must understand what the researchers did, identify the procedure needed to test the claim, and carry out that procedure as an AI agent. This matters because papers often describe results more clearly than they describe every practical step required to obtain them. Replication therefore tests more than reading comprehension. It tests whether an agent can turn written research into working experimental work.
Inherent evaluates this ability through the Replica task framework, introduced in the team’s arXiv paper. The benchmark uses held-out replication tasks, so the evaluation focuses on tasks reserved for testing rather than examples used to guide development. An auto-generated rubric judge scores the results against defined criteria. That gives the benchmark a repeatable way to assess whether an attempted replication actually meets the requirements of the task.
Faraday is a 27-billion-parameter AI Scientist agent. According to the paper, it surpassed Claude Opus 4.8 and GPT-5.5 on these held-out replication tasks. The important point is not simply that a smaller model scored higher. It is that Faraday was built and measured around a workflow that extends from a paper’s claims to experimental execution.
Replication also serves a larger purpose. Recovering the work that papers leave implicit can reveal what is needed to move from an existing result toward a new idea. The takeaway is simple: Faraday treats research as something to perform, not just something to summarize.
How a 27B model beat larger closed systems
Faraday’s result challenges a common assumption about research agents: that the largest model will produce the strongest scientific work. Inherent says its agent, built on Qwen 3.6 with 27 billion parameters, outperformed Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5 when asked to replicate findings from published papers.
The surprising part is that Faraday was not given the answer in advance. It had to understand the experimental procedure and carry it out well enough to reproduce the reported result. Inherent says Faraday achieved a higher success rate than the larger frontier-scale systems in this task.
The company attributes the difference to reinforcement learning. Rather than training the agent only to generate plausible responses, Inherent trained it to make better research decisions. This includes developing what the company calls “research taste”: an instinct for which experiments to design, which results to prioritize, and how to direct effort toward the most useful next step.
That distinction matters because scientific replication is not a simple question-answering task. A model can know many facts and still choose a poor procedure, waste effort on low-value tests, or miss the experiment most likely to clarify the result. Faraday’s performance suggests that the quality of those choices may matter more than raw model size for some research work.
The result does not show that smaller models are always better, or that Faraday is stronger at every task. It shows something narrower and more useful: a focused agent with the right training can outperform much larger systems on a demanding, practical benchmark. For research automation, decision-making may matter as much as knowledge.
What the 310-task Replica benchmark measures
Most AI evaluations ask whether a model can produce an answer. Replica asks a harder question: can an agent independently reproduce the findings reported in scientific papers?
The benchmark contains 310 research replication tasks. Each task is built around published research, so success is not just a matter of recalling facts from training data. The agent must work from the paper and carry out enough of the research process to test whether its findings can be reproduced. The available research description does not specify the exact mix of papers, fields, datasets, or scoring rules, so the safest interpretation is narrow: Replica measures performance on independently reproducing published research findings.
That distinction matters. A coding model can write syntactically correct code and still fail to recreate a paper's result. Scientific replication also requires understanding what the paper is claiming, identifying the relevant method, using the right data or experimental setup, and checking whether the output supports the original conclusion. Faraday was designed for this broader task. Inherent describes it as a 27B-parameter model that adds scientific intuition to coding agents and was trained with long-horizon reinforcement learning.
The benchmark therefore tests more than programming speed. It tests whether an agent can sustain a research workflow long enough to connect a paper's description to a working reproduction. Inherent says Faraday outperformed Claude Opus 4.8 and GPT-5.5 on replicating research papers, but the supplied material does not provide the benchmark scores or evaluation breakdown.
The key takeaway is simple: Replica measures research execution, not answer generation. A model must produce evidence that a published result can be recreated, not merely explain what the paper says.
Why held-out results matter more than familiar tasks
A model can look capable when it succeeds on tasks that resemble its training examples. That is why results on familiar coding problems are not enough to show that an AI scientist can reproduce research. The stronger test is whether it can handle papers it has not encountered during training, using the methods and evidence in those papers rather than recalling a learned pattern.
Faraday is described as a 27B-parameter AI Scientist trained with long-horizon reinforcement learning. Its reported target is research replication, not ordinary code completion. That distinction matters because replication requires an agent to follow a paper’s setup, make many connected decisions, run experiments, inspect results, and respond when something fails. A familiar benchmark may reward a short correct answer. An unfamiliar paper tests whether the system can carry a research process through to a result.
This is also why comparisons with Claude Opus 4.8 and GPT-5.5 are meaningful only when the evaluation separates known material from held-out material. If the papers were familiar to a model, success could partly reflect exposure. If they were held out, stronger performance is better evidence that the agent learned useful scientific behavior.
The result still needs careful interpretation. Replicating a paper is valuable, but it is not the same as producing novel research. Replication tests whether an agent can reconstruct an existing result. Novel research also requires choosing worthwhile questions and deciding which ideas deserve pursuit.
The takeaway is simple: the best evidence is not that an agent solves a task it has seen before. It is that the agent can reproduce a result from an unfamiliar paper, with little help, and make its way through the work.
How GPT-5.5 Codex fits inside Faraday's workflow
GPT-5.5 Codex is not presented as Faraday’s foundation or hidden operator. It is one of the frontier agents used to measure what Faraday can do. In the replication benchmark, Faraday, a 27-billion-parameter agent, outperformed both GPT-5.5 Codex and Claude Opus 4.8.
That result is unexpected because the task is not simply writing code. Replicating a research paper requires an agent to understand the paper’s claim, decide which details matter, build an experiment, run it, inspect the results, and respond when the first attempt fails. Coding is necessary, but it is only one part of the work. Scientific judgment determines what to test and whether the result actually supports the paper.
Faraday was trained with long-horizon reinforcement learning. Coding agents serve as tools during that training, giving Faraday a way to carry out experiments rather than stopping at a written plan. GPT-5.5 Codex therefore sits on the comparison side of the workflow, while coding-agent capabilities represent one of the tools Faraday learns to use. The available research does not say that Faraday directly calls GPT-5.5 Codex during each task.
This distinction matters for system design. A strong coding model can produce an implementation, but an AI scientist must also choose useful questions, connect evidence to claims, and keep working across a long sequence of decisions. Faraday’s advantage suggests that performance comes less from model size alone and more from training an agent to manage the whole research process.
The takeaway is simple: Codex is a powerful benchmark for coding ability, while Faraday is trained to place coding inside a larger loop of scientific reasoning and verification.
Can stronger tools improve an agent without retraining it
A larger model is not the only way to improve an AI research agent. The comparison between Faraday, Claude Opus 4.8, and GPT-5.5 suggests that the surrounding system matters too. Claude and GPT-5.5 were tested through their standard Claude Code and Codex harnesses, with thinking effort set to extra high. Faraday still produced more faithful replications across every paper category in the task suite.
That result is important because the models were not being judged on memorized answers. They had to reproduce research, including work from held-out AI-for-science domains. Faraday also struggled less with recent research, which indicates that its scientific ability was useful beyond material seen by the base model during pre-training.
The surprising part is not simply that a smaller model won. It is that an agent can gain practical capability from how it works, not only from how many parameters its base model has. The harness, tools, task process, and scientific routines can affect whether the model turns information into a reliable experiment.
This does not prove that tools alone explain Faraday’s advantage. The available results do not isolate each component or show that retraining was unnecessary. They do show that frontier-scale models, even with high thinking effort and established coding harnesses, did not saturate the replication task space. Faraday performed especially well in meta-learning, structural biology, and materials science, including domains held out from training.
The practical lesson is clear: improving an agent may involve building a better research workflow around a model, not just replacing the model with a larger one. Stronger tools can matter, but their value must be measured by the quality of completed research, not by model size or effort settings alone.
Where automated rubric judges help and where they can mislead
A research agent needs to be judged on more than whether its final number matches a paper. Faraday’s test also asked whether it showed “research taste”: choosing worthwhile experiments and designing them well. A rubric can make those expectations visible. Instead of giving credit only for a correct result, evaluators can check whether the agent selected sensible methods, used the available evidence, and produced a result that actually supports the paper’s claim.
That is useful because scientific replication has several ways to fail. An agent might reach a similar conclusion through an invalid shortcut, or produce technically correct code that answers the wrong question. A structured score helps separate these cases. It also makes comparisons between systems more consistent, especially when Faraday is measured against much larger models such as Claude Opus 4.8 and GPT-5.5.
The danger is that a rubric can make judgment look more objective than it is. A judge can score what the rubric names, but it may miss an important experiment that the rubric did not anticipate. This matters especially for “research taste,” which is partly about noticing what deserves investigation in the first place. A system could learn to satisfy visible scoring rules without becoming better at scientific reasoning.
There is also a difference between reproducing a published result and discovering new knowledge. Replication has a known target, even when the agent is not shown the answer in advance. Discovery does not offer that same reference point.
The practical lesson is simple: automated judges are useful measurement tools, not final authorities. They can make research agents easier to compare, but the rubric must not become a substitute for scientific judgment.
What replication reveals about building AI scientists
Replication is a useful test because it asks an agent to do more than produce plausible prose. On Replica, the agent must reconstruct missing scientific results under fixed compute and time limits. The benchmark contains 310 tasks focused on reproducing scientific figures. Faraday, Inherent’s 27B research agent, led GPT-5.5 and Claude Opus 4.8 on these tasks.
The surprising lesson is that an AI scientist is not defined by model size alone. Faraday’s performance points to the importance of post-training and tool orchestration: how the agent is taught to work, which tools it uses, and how it organizes its investigation. Replication makes those choices visible because the output must match an existing result, not merely sound reasonable.
Inherent’s design also reflects a specific idea of scientific taste. The goal is not to build an agent that agrees with the user or produces the answer the user expects. Hughes described the preferred teammate as someone who becomes curious, runs additional experiments, and returns with results for discussion. That behavior changes the workflow from asking for an answer to reviewing evidence produced by an active collaborator.
The company has also chosen not to build every component itself. Faraday uses OpenAI’s GPT-5.5 Codex as its coding tool, much as human scientists rely on existing software instead of writing all their own tools. This division of labor keeps the agent focused on research judgment rather than recreating basic infrastructure.
Replication therefore reveals a practical blueprint: train for investigation, give the agent reliable tools, and reward findings that can be checked. The strongest AI scientist may be less like an all-knowing oracle and more like a careful colleague who knows when to test an idea.
What engineering teams can learn from Faraday's design
Faraday’s result suggests that agent performance depends on system design, not only model size. On Replica, the agent had to reconstruct missing scientific results under fixed compute and time limits. That meant inferring what to run, writing and executing code, and reproducing data without seeing the original figure. The task rewarded useful decisions over isolated language ability.
The surprising part is that a 27-billion-parameter supervisory model, post-trained from Qwen-3.6-27B, could compete with larger general-purpose systems on this narrow workflow. Its role was not simply to generate an answer. It supervised the research process and separated planning from implementation. That separation gives the system a place to decide what work matters before spending resources on execution.
This design has a direct lesson for engineering teams building long-running research or coding agents. A single model call is poorly suited to a task that requires selecting experiments, writing code, checking results, and managing a limited budget. A supervisory layer can coordinate those steps, while tool use turns decisions into executable work. Post-training can then focus the model on the habits the workflow rewards, such as planning, iteration, and scoping.
Replica’s 310 figure-replication tasks, drawn from 100 machine learning and AI-for-science papers, also point to a useful development strategy. Reproducing existing results gives agents practice in choosing experiments and developing research taste before they are asked to propose novel hypotheses.
The takeaway is practical: build agents around the work loop, not just the model. Specialization, supervision, tools, and budget awareness can matter more than adding parameters.