Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks
Summary
Anthropic’s Fable 5.2 scored 41.3% on Scale AI’s Remote Labor Index, ahead of the next-ranked model at 21.25%. The benchmark evaluates complete freelance projects across 23 categories, comparing AI deliverables with professional human references through trained human review. A project passes only if its output meets the brief at least as well as the reference and would be acceptable to a reasonable client. Most Fable projects did not pass. The test covers varied file-based work but excludes tasks requiring client interaction, teamwork, physical labor, or outcomes that take time to assess. Anthropic identified the run as high effort with safeguards enabled, but did not disclose exact execution costs or human support.
