The kovara CLI

kovara is the open-source (Apache-2.0) command line for compute repositories: write and validate kovara.yaml, benchmark an OpenAI-compatible endpoint, publish manifests and runs, and pull published manifests. Source: packages/kovara-cli/.

Install

From a checkout of this repository:

pip install ./packages/kovara-cli
kovara --version

Requires Python 3.9 or newer; the only dependency is PyYAML. Offline, with setuptools already installed: pip install --no-build-isolation ./packages/kovara-cli. Without installing: PYTHONPATH=packages/kovara-cli/src python -m kovara ….

Configuration

Exit codes: 0 success · 1 failure (invalid manifest, HTTP or network error, refused publish) · 2 usage error · 130 interrupted.

The CLI never prints an API key: keys are shown as their public prefix (kvk_1a2b3c4d…). It does not follow HTTP redirects from the Kovara API (a redirect could carry the key to another host); pass the final address with --api-url.

Commands

kovara init

Writes a new kovara.yaml. In a terminal it asks for anything required that is not given as a flag; with --no-input (or without a terminal) missing values are an error. Refuses to overwrite an existing file without --force. The result is validated before it is written.

kovara init --model Qwen/Qwen3-32B --model-revision <commit sha> --engine vllm --engine-version 0.11.0 \
  --accelerator nvidia-h200-sxm --gpu-count 2 --precision bf16 --tensor-parallel-size 2 --max-model-len 8192

Flags: -o/--output (default kovara.yaml), --force, --no-input, --kind, --model, --model-source (huggingface or other), --model-revision, --engine, --engine-version, --image, --accelerator, --gpu-count, --precision, --quantization, --tensor-parallel-size, --max-model-len, --profile, --concurrency, --input-tokens, --output-tokens, --notes. Inference kinds get a benchmark section (profile interactive-chat, concurrency 1,8,32 unless given).

kovara validate [path]

Validates a manifest (default ./kovara.yaml) with the rules in kovara-yaml.md and prints its revision and content hash. Errors go to stderr, one per line as path: message, and the exit code is 1. Warnings (unpinned model revision or image) do not fail validation.

$ kovara validate
kovara.yaml is valid (schema_version 1, inference)
revision      r0b7cbeafb9f8
content_hash  0b7cbeafb9f857aeef2612b8058e7a895d66370a2df3722f0c1c16a7ffa2af7b
warning: model.revision: model.revision is not set. Pin the model to a commit SHA so others can reproduce the workload

--json prints the manifest's canonical JSON instead (what the web app accepts when you paste a manifest).

kovara inspect [path]

Summarizes a manifest, or a results file written by kovara benchmark: run kind, conditions, a table of every concurrency level (successes, errors, timeouts, throughput, TTFT/ITL/latency percentiles), and the local-only details (endpoint, failures, warnings).

kovara benchmark

kovara benchmark --endpoint http://localhost:8000 --manifest kovara.yaml \
  --concurrency 1,8,32 --requests 100 --warmup 4 --out results.json

Progress and warnings go to stderr. The method is described in benchmarking.md.

kovara login / kovara logout

kovara login                       # prompts for the key (hidden input)
echo "$KEY" | kovara login         # or from stdin
KOVARA_API_KEY=kvk_… kovara login  # or from the environment

Create keys on your Kovara account page (Developer settings) with the scopes you need: repos:write to publish manifests, benchmarks:write to publish runs, repos:read to pull private repositories. login checks the key with GET /api/v1/whoami and stores it in credentials.json (directory mode 0700, file mode 0600), one entry per Kovara address. logout removes the entry (--all: every entry). Removing a key locally does not revoke it; revoke it on the website if it may have leaked.

kovara publish manifest [path] --repo OWNER/SLUG

Validates the manifest, shows the exact JSON that will be sent ({"manifest": …, "notes": …}), asks for confirmation (--yes to skip; required when not in a terminal) and creates a new immutable version. Prints the version number, revision and the repository's versions page, and checks that the server's content hash matches the local one. Publishing the same manifest twice is refused by the server (HTTP 409). --notes adds release notes (≤ 2000 characters).

kovara publish results FILE --repo OWNER/SLUG [--revision rXXXXXXXXXXXX]

Publishes the runs of a results file to the repository version they were measured against: --revision, else the revision of the manifest used for the benchmark (recorded in the file), else the repository's latest version (with a warning). Refuses incomplete files, invalid runs, and runs carrying anything outside the run record (the local section is dropped; URLs are refused). Shows the payload, asks for confirmation, sends up to 16 runs per request and prints each run's URL and evidence state.

kovara pull OWNER/SLUG[@REVISION] [-o kovara.yaml]

Downloads a version (@rXXXXXXXXXXXX, or the latest), verifies locally that the manifest hashes to the server's content_hash and that the revision matches it, then writes deterministic YAML (-o - for stdout; --force to overwrite). Public repositories need no key; a stored key is sent when present (and the request is retried without it if the key is refused). Nothing is written when any check fails.

Examples

packages/kovara-cli/examples/ holds valid manifests for Qwen3-32B on two H200 SXM with vLLM, Llama 3.1 8B Instruct on one H100 SXM with SGLang, and Llama 3.1 70B Instruct on one MI300X with vLLM. They leave model.revision (and, for SGLang, runtime.version) for you to fill in with the values you actually run; kovara validate warns until the model is pinned.

Testing the CLI

python -m unittest discover -s packages/kovara-cli/tests -v

The tests run local mock servers on 127.0.0.1 (an OpenAI-compatible streaming server with configurable delays, usage on/off, error injection and stalls; and a stand-in for the Kovara API). Results measured against the mock are protocol tests. python packages/kovara-cli/tests/mock_server.py starts the mock by hand for experiments.