Community Trust ScoreVerified
Thirteen AI agent configurations tested by UniPat AI showed stronger agreement with one another than with measured experimental outcomes, according to a study submitted to arXiv on Nov. 11. The research, titled “Science Prediction Universe,” evaluated how well these systems could rank experimental results across chemistry, materials science, biology, and medicine.
More coverage is filed under Technology.
Correlation Gap
The study examined 13 agent configurations built from 12 different models, each combining a specific model, reasoning setting, and agent software. The agents were tasked with ranking specified experiments by metrics such as yield or conductivity, without attempting to predict specific numerical values. A perfect ordering that matches measurements scores 1, while random rankings average 0.
On 100 public tasks, the average agreement between the scored configurations reached a Spearman rank correlation of 0.52. In contrast, the average agreement against the actual measured outcomes was 0.33. This discrepancy was consistent across the majority of the test set, with agent agreement exceeding their correlation with measurements on 81 of the 100 public tasks. The authors note that these figures describe how closely the orderings match, rather than the percentage of answers that were correct.
Pooling the predictions from the different systems produced a consensus ranking that scored 0.08 points above the average individual submission. This consensus approach improved results on 73 of the 100 public tasks, with a 95% confidence interval for the gain ranging from 0.05 to 0.11 points. The authors state that averaging can reduce errors that differ between systems while retaining common ones, suggesting that shared errors are connected to simple assumptions about how experimental outcomes should rise or fall.
Private Set and Model Limitations
The benchmark included 16 private tasks across four unpublished experimental series. Of 208 official private runs, 159 were scored. Twelve of the 13 configurations ranked the unpublished experiments significantly better than chance overall. However, the overlap in scoring across the configurations was limited. All 13 configurations shared just 28 scored tasks, comprising 27 public tasks and one private task.
Specific models faced significant limitations in the public set. Claude Opus 5.5 had scores on only 44 public tasks, while Claude Fable 5.1 had scores on only 30 public tasks. The study attributes these low counts largely to safety refusals by the models. Despite these limitations, the benchmark’s best individual system beat the consensus of its peers on matched tasks. Additionally, tasks with greater agreement among agents tended to have better predictions.
The gap between agent agreement and reality was stark in specific cases. On one oncology task, eleven configurations agreed at a score of 0.9, yet their average score against the actual measurements was just 0.27. The public tasks looked backward at published measurements and were selected after pilot evaluations. The study used no human prediction baseline, and the authors emphasize that lab measurements are still needed to check the answers. The results were released on Nov. 12.
