AI models agreed with each other more than with reality in a new test
A study tested 13 AI agent configurations on experimental-ranking tasks in chemistry, materials science, biology and medicine. On public tasks, systems agreed more with each other (average rank correlation 0.52) than with measured outcomes (0.33), and agreement exceeded measurement accuracy on 81 of 100 tasks. Averaging rankings improved scores by 0.08 on average, but shared errors persisted; on one oncology task, 11 systems agreed strongly while matching measurements poorly. Most systems performed above chance on a small set of unpublished experiments. The study cautions that results cover limited tasks, systems had uneven scoring coverage, and no human baseline was included. Consensus can help, but laboratory measurements remain necessary.
