A benchmark run on Kovara is a claim: "this workload, on this hardware, under these conditions, measured these numbers." Kovara shows every run with how much it can be trusted, and never mixes runs that cannot be compared.
Evidence states
| State | Meaning | How a run gets there |
|---|---|---|
self_reported | Published by its author; nobody else has confirmed it. | Default for every run. |
reproduced | Another person published a comparable hardware run of the same repository version. | Automatic, in the database, when the second comparable run arrives. Both runs move up. |
independently_verified | Kovara checked the evidence behind the run. | Only Kovara's evidence administrator can set it (kc_benchmark_verify). |
artifact_backed | Reserved for runs with stored raw artifacts (per-request timings, server logs). | Not available in this release. |
States only move up: a reproduction never downgrades a verified run. A withdrawn run (by its author or a repository writer) or a run hidden by moderation drops out of counts and comparisons; its history is kept.
What the states do not mean: reproduced says two people measured similar conditions and published, not that their numbers agree or that either is correct; the CLI cannot see the hardware behind an endpoint, so the accelerator of a hardware run is what its manifest states.
Comparability
Two runs are comparable when all of these are equal (the database's comparability_key, the SHA-256 of their canonical JSON):
- model id and model revision
- engine and engine version
- accelerator and GPU count
- precision
- profile, input tokens and output tokens
- concurrency
- run kind
Runs that differ in any of them are shown side by side at most with the difference stated, never averaged or ranked together. Things that are recorded but not part of the key — request count, warmup, timeout, the client's Python and OS, token_count_method, ignore_eos_honoured — are shown with each run; check them before drawing conclusions. In particular, compare throughput only between runs whose output_tokens_mean reached output_tokens, and don't compare stream_chunks counts with server_usage counts.
Protocol tests
Runs with run_kind: "protocol_test" test tooling against a mock or controlled endpoint. They are listed separately, never compared, never reproduced or independently_verified, and never used for prices or economics.
Predictions are not measurements
Repository pages may show RA-BIT's analytical estimate next to measured runs. It is always labelled as a prediction and never counts as evidence for a run.