Computational research often comes with code to review. But reviewers don't always have the time to check whether claims reproduce from the code.
Reviewer3 has been developing Code Replication: a new approach to review code that uses AI to retrieve code in a linked repository and run it in an isolated execution environment. We then check manuscript claims against the run's results and flag any issues. It works on papers with linked code or supplemental code files. No additional author or reviewer preparation is required.
By default, we work across a wide range of scientific disciplines, and our sandbox comes with nine language runtimes preinstalled. We support common research repositories and data-sharing services, including GitHub, GitLab, Bitbucket, Codeberg, Zenodo, Figshare, Dryad, and Dataverse. Execution is CPU-only. If you have other requirements, we can work with your publication to develop code replication support for your workflow.
How it works
1. Extract and run the code
We start by reading the manuscript to identify linked code. You can also upload code directly as a supplemental zip file. If a repository can't be accessed programmatically, or we hit errors during retrieval, we'll prompt you to upload the files manually. Our goal is to work with the code authors have already provided, rather than asking them to prepare it in a particular format.
We then open an isolated execution environment and run the code, recording what we attempted, what ran, and what the results were. The first question we can answer is whether the code runs at all. Does the repository have everything it needs, or did something block Reviewer3 from running it?
2. Replicate manuscript claims
Next, we identify scientific claims that depend on code. These might include a reported model accuracy, a correlation coefficient, or a value in a results table. For each claim, we'll compare the value the code produced with the value reported in the manuscript, and flag any inconsistencies.
For Reviewer3 to mark a claim as verified, a fresh execution must produce the reported result. We don't count cached notebook outputs, saved logs, rendered figures, results files, or numbers quoted in a README as independent verification. If we can't run the code and produce the result ourselves, we report that we couldn't check it and the underlying cause.
Every claim receives one of four verdicts:
| Verdict | Meaning |
|---|---|
| Verified | A fresh execution matched the reported result within the paper's stated tolerance. |
| Mismatch | A fresh execution produced a result that contradicts the paper. |
| Unsure | We obtained a fresh result, but the comparison is inconclusive. |
| Could not check | We couldn't obtain a fresh result. |
3. Read the code files
We'll also read the code files themselves and check for issues that running the code wouldn't surface. Reproducing the reported numbers is only part of the problem. A paper might say it used patient-level cross-validation, when the code actually performs a random shuffle. Running the code might still produce the reported results, but there is still a code-related issue, as the analysis isn't sound or performing as expected.
We look for these discrepancies even when the code successfully reproduces the paper's findings, and report any issues separately. This gives reviewers information about both the results and the methods used to obtain them.
4. Displaying results
Code Replication findings appear in their own section of the Reviewer3 report, rather than in the general review tab. They can be run alone or alongside other Reviewer3 manuscript checks.
Each finding includes the claim being checked, the relevant sentence from the manuscript, the observed value, the difference between the reported and observed results, and an explanation of the discrepancy.
These findings don't receive the severity labels and are never classified as fatal flaws. The purpose is to give editors and reviewers the evidence they need to evaluate the manuscript themselves.
Languages supported
| Language | Version |
|---|---|
| Python | 3.12 |
| R | r-base, Debian bookworm |
| JavaScript (Node) | 20 |
| Julia | 1.10.4 |
| Go | 1.22.5 |
| Rust | Stable, via rustup |
| C / C++ | gcc, g++, cmake |
| Fortran | gfortran |
| Java | JRE only |
If a repository requires additional dependencies, the execution session attempts to install them. We don't support licensed software such as MATLAB, Stata, or Gurobi, at this time.
Execution timeline
Some papers take only a few minutes to reproduce, but some take longer, particularly if they involve model training or large datasets. We use an execution limit of 4 hours and will report a log of what was attempted in that timeframe along with any failures blocking replication.
Try Code Replication
Code Replication can run independently of our other manuscript checks.
If you're a publisher or journal interested in evaluating the reproducibility of submitted research, you can pilot Code Replication on a batch of manuscripts without enabling any of our other reviewers.
Email us to set up a pilot.