Publishing manifests and benchmark runs

Before you start

1. Sign in on Kovara and choose a username (your handle): repositories live at /repos/<handle>/<slug>. An organization owner can give the organization a handle too; its members can then publish under it.
2. Create a repository on the website (/repos/new).
3. Create an API key (account page, Developer settings) with the scopes you need — repos:write for manifests, benchmarks:write for runs, repos:read to pull private repositories — and run kovara login. The key is shown once; Kovara stores only its SHA-256.

Publish a manifest version

kovara validate kovara.yaml
kovara publish manifest kovara.yaml --repo acme/qwen3-32b-h200 --notes "vLLM 0.11.0, TP 2"

The CLI shows the JSON it will send and asks before sending. The server validates it again (the same rules, plus: the accelerator must be a GPU in Kovara's catalogue), computes the canonical JSON and SHA-256, and stores an immutable version: version_number (1, 2, …), revision (r + 12 hex digits of the hash) and content_hash. The CLI checks the server's hash against its own. Publishing a manifest identical to an existing version of the repository is refused (HTTP 409); change a value in the manifest (its own notes field counts; release notes given with --notes are not part of the manifest or its hash) to publish a new version.

Versions never change after publication. To fix a mistake, publish a new version.

Publish benchmark runs

kovara benchmark --endpoint http://localhost:8000 --manifest kovara.yaml --out results.json
kovara publish results results.json --repo acme/qwen3-32b-h200

Runs are filed under the manifest version they were measured with: the revision recorded in the results file (the manifest given to kovara benchmark), or --revision. Publish the manifest first so that revision exists; otherwise the server answers "Version not found". The server also checks that each run's model id and engine (and, for hardware runs, accelerator) match that version's manifest.

What is sent: {"revision": "r…", "runs": [run, …]} with up to 16 runs per request (larger files are split). Each run is exactly the record described in benchmarking.md. Never sent: the local section (endpoint address, host, served models, error messages), prompts, responses, keys. The CLI refuses to send incomplete results, invalid runs or anything URL-like, and shows the payload first.

Anyone who can read a repository may publish runs against it — reproductions by other engineers are the point. Runs start as self_reported; see evidence.md. The same exact result can be published only once.

Limits

Over a limit the API answers HTTP 413 (size) or 429 (rate).

HTTP API

The CLI uses these endpoints (all JSON; keys as Authorization: Bearer kvk_…):

Errors are {error} with status 400, 401 (key invalid, revoked or expired), 403 (missing scope or access), 404, 409 (duplicate), 413, 422 (validation), 429 (rate limit) or 503.

Withdrawing and reporting

A run's author, or a writer of its repository, can withdraw it with a reason; it then leaves counts and comparisons. Anyone signed in can report a repository, run or profile (spam, fraudulent benchmark, abuse, confidential data, ownership); Kovara staff review reports and can hide content.

If you published a credential by mistake, revoke it with its issuer immediately, then withdraw the run or report the version (published versions cannot be edited).