analyst bench
An evaluation harness that scores whether a model can do entry-level analyst work on real filings.
Nobody could say whether a model was fit to do the work of an entry-level analyst. Demos answered easy questions from clean text; the actual job — reading a filing against everything else the company has disclosed, then defending the number — went untested. Every decision came down to a bake-off nobody could reproduce.
A 927-question evaluation set drawn from filings and expert judgment, split into public, validation and held-out test so scores cannot be trained against. An agent harness gives each model six research tools and a two-hour ceiling per task, and grading is dealbreaker-gated: miss a critical detail and the answer scores zero, however well the rest reads. Three independent judges score every response.
Model selection became a measurement instead of an argument. Nine analytical categories are scored separately, so a model that reads disclosure well and models cash flow badly shows up as exactly that — and because the test split stays held back, the number still means something a release later.