Benchmark method

kovara benchmark measures your own OpenAI-compatible endpoint. Kovara runs nothing on its side: no GPUs, no inference, no execution of manifests. This page explains exactly what the CLI measures, how, and what it does not.

What a benchmark does

1. Checks the endpoint. GET /v1/models must return an OpenAI-style model list. The model to request is --model if given, otherwise the manifest's model.id, which must be in that list (for a protocol test without a manifest: the only model served). A hardware run whose requested model differs from model.id (an alias passed with --model) is recorded under model.id with a warning.
2. Sends one small streaming request to POST /v1/chat/completions with stream: true, stream_options: {include_usage: true} and ignore_eos: true. If the server answers 400/422, it retries without ignore_eos, then without stream_options, and records what was accepted.
3. Calibrates the prompt length when the server reports token usage: it sends up to four 1-token requests and adjusts the number of words until the server's prompt_tokens (which include the chat template) are within 1% (at least ±2 tokens) of input_tokens. Without usage, prompts use one word per requested token and the actual prompt length is unknown.
4. Warmup: --warmup requests (default 4) with up to the highest concurrency level in flight. Not measured.
5. For each concurrency level (default 1,8,32): --requests requests (default 100), with that many in flight. Each in-flight slot is a thread with its own keep-alive connection, opened before the level starts so connection setup is not counted in the first request. A thread starts its next request as soon as its previous one ends.
6. Writes a results file with one run record per level (the shape of ARCHITECTURE.md §5) and a local section.

Requests use temperature: 0, max_tokens = output_tokens and, where accepted, ignore_eos: true so every completion should run to output_tokens.

Prompts

Synthetic and deterministic: an instruction, the request's number, then words drawn from a fixed list of common English words by a seeded generator (--seed, default 7; generator id kovara-synthetic-v1). The same seed, index and length give the same prompt on every machine. Every request has its own number and words, so no two requests share more than the short instruction (prefix caching cannot turn the benchmark into cache hits). Prompts contain no private data and are never published.

What is measured

All times use time.perf_counter() (monotonic, high resolution). The stream is read line by line as it arrives (Server-Sent Events); a chunk's time is the moment its line was read from the socket.

A request succeeds when the server answers HTTP 200, the stream ends with [DONE] or a finish_reason, it carried at least one content chunk, and no error event arrived. Timeouts: the whole request (connect, headers, stream) exceeded --timeout seconds (default 120). Everything else is an error (HTTP status, connection failure, error event, malformed stream, stream ended early, empty completion). Only successful requests enter the latency distributions and throughput. Error messages are kept in the results file's local section only.

Token counting

• token_count_method: "server_usage" — every successful request reported usage.completion_tokens (the final chunk with stream_options.include_usage). Output and input tokens are the server's counts.
• token_count_method: "stream_chunks" — at least one successful request had no usage. Output tokens are then the number of streamed content chunks, which equals tokens only when the server streams one token per chunk; input tokens are not measured (null). The label travels with the run.

The CLI never estimates tokens from characters or words.

environment.ignore_eos_honoured is true when every successful request returned exactly output_tokens completion tokens, false when some returned fewer (the server accepted ignore_eos but stopped early, or ignored it), null when it could not be checked (no usage, or ignore_eos not sent). Compare metrics.output_tokens_mean with output_tokens before comparing throughput between runs.

Percentiles

Linear interpolation between the closest ranks of the sorted sample (numpy's default linear method, Hyndman & Fan type 7): for n sorted values and quantile q, h = (n − 1)·q, value = x[⌊h⌋] + (h − ⌊h⌋)·(x[⌊h⌋+1] − x[⌊h⌋]). Recorded as environment.percentile_method: "linear".

A percentile is reported only when the sample supports it: p50 from 1 sample, p95 from 20, p99 from 100. Each distribution carries its sample size n; smaller samples leave the higher percentiles out (never the maximum dressed up as a tail). With the default 100 requests per level, TTFT and latency get a p99 only if all 100 succeed; ITL samples are per chunk and usually far more numerous. Values are rounded to 0.001 ms, rates to 0.001 per second.

Protocol tests and hardware runs

The CLI cannot see the hardware behind an endpoint: a hardware run states what the manifest says the endpoint runs on. That is why runs start as self_reported (see evidence.md).

Interrupting

Ctrl-C stops the benchmark: in-flight connections are closed, completed levels are kept, the interrupted level's partial measurement is stored under local.partial_run, and the file is marked "complete": false. Incomplete results cannot be published. The exit code is 130.

The results file

{
  "format": "kovara-benchmark-results", "format_version": 1, "complete": true,
  "created_at": "2026-10-29T12:00:00Z",
  "manifest": {"revision": "r…", "content_hash": "…"},
  "runs": [ { "run_kind": "hardware", "model_id": "…", "concurrency": 1, "metrics": {…}, "environment": {…}, … } ],
  "local": { "endpoint": "http://…", "endpoint_host": "…", "served_models": […], "levels": [{"errors": {…}}], "warnings": […] }
}

runs hold exactly the published record. environment holds the CLI version, Python version, OS and machine architecture of the client, sampling settings, ignore_eos_honoured, the prompt generator, the percentile method, the model name the server reported (server_model), stream and the client timeout — no host names. local never leaves your machine.

What is published and what is not

Published (per run): the conditions (model id and revision, engine and version, accelerator, GPU count, precision, profile, token lengths, concurrency, request and warmup counts), the metrics above, the environment described above, and the run kind.

Never published: prompts, responses, the endpoint URL, host names or IP addresses, the endpoint's API key, error messages from the server, the list of models the server serves. kovara publish results sends only runs (and the revision), refuses runs with unknown fields or anything URL-like, and shows the exact payload before asking.

Reproducing someone's run

1. kovara pull owner/slug@rXXXXXXXXXXXX -o kovara.yaml — the exact manifest version, hash verified.
2. Serve the same model revision with the same engine version (and image digest, if the manifest has one) on the same accelerator and GPU count, with the manifest's precision and parallelism.
3. Run with the same conditions as the published run: kovara benchmark --endpoint … --manifest kovara.yaml --profile P --concurrency C --input-tokens I --output-tokens O --requests N --warmup W --seed S (all shown on the run's page and in its record).
4. kovara publish results results.json --repo owner/slug --revision rXXXXXXXXXXXX. When another person publishes a comparable hardware run of the same version, both runs become reproduced.

Comparable means the same model id and revision, engine and version, accelerator, GPU count, precision, profile, input and output tokens, concurrency and run kind (the database's comparability_key).

Limitations

• The client machine and network are part of the measurement. Run the CLI close to the endpoint (same host or network) for numbers that describe the server rather than the path to it.
• One Python process drives all in-flight requests with threads. Python's global interpreter lock and thread scheduling add small delays at high concurrency (hundreds of streams) and very short inter-chunk gaps; the results label the client (environment.python, machine) so this can be judged. Prefer several shorter runs to one extreme level.
• Timestamps are taken when a line is read from the socket. Proxies or servers that buffer the stream make TTFT and ITL describe the buffering.
• The CLI does not know the endpoint's hardware, configuration, other tenants or background load.
• Without server usage, the prompt length is not calibrated and token throughput is chunk throughput (labelled stream_chunks).
• Requests are closed-loop (a new request starts when one ends); open-loop arrival rates are not modelled.
• Only /v1/chat/completions with streaming is benchmarked.