Evaluating work, not chat

The comfortable metric

Most public evaluation measures how an answer reads: fluent, confident, plausibly correct. It is the metric the field reaches for because it is cheap to run, easy to chart, and because reading well is what a language model is already good at. But users do not read answers — they use them. An intelligence can write a flawless paragraph about a repository and change nothing in it; it can also say almost nothing and land the fix. Between those two, only one did the work, and the comfortable metric cannot tell them apart.

An answer that reads well and changes nothing is a well-written failure.

Work has an end state

The field grades prose because prose is what most of it ships. When the product is work, the evaluation gets harder — and more honest. A task either completes or it does not. The tests pass or they fail. The pull request is open or it is not. There is no judge to charm, no rubric to game, no vibe to average across a hundred runs until the number looks kind.

That is the whole reason we chose it. An observable end state is a fact, and a fact does not care how the attempt was phrased.

What we actually measure

The program evaluates completed tasks, not sampled chat. A repository is read. A test suite is run. A change is proposed; a pull request is opened and defended on its diff. Each task carries a single observable end state, and that is what makes the result repeatable: the same task, through the same frozen harness, produces a count that means the same thing next quarter as it does today — checked by us first, and in time by anyone who wants to.

Connecting evaluation to product proof

The program proposes using the same task shapes in evaluation and in a future released Build surface. That link must be demonstrated at the real product boundary; this page does not establish that repository work or pull-request completion is publicly available.

The evaluation should follow the product boundary, never replace it.

A result is not a claim

Inside Nova Intelligence, a strong run does not promote itself into a sentence on a marketing page. Every outward claim waits behind its evidence gate, and a flattering number does not get to jump the queue. We would rather say less than the results suggest than one word more than they prove — because the first falsifiable overstatement is the one a curious user tests first.

What we will publish, and when it fails

The task set, the harness, and the protocol go on the record before the run — the same pre-registration discipline as the rest of the program — and the results are published with the failures kept in frame beside the successes. A metric that can only move up is a marketing instrument. Ours is built to be allowed to move down.

Nothing on this page is a score. It is a description of how the scores will be taken, published so the counting can be checked before the counting begins.