Track training, fine-tuning and evaluation runs: what ran, with which settings, on which GPU, how the metrics moved, and what it cost. Runs can be compared side by side, and the models and datasets they produce and use are kept in a registry.
Open it at Experiments (/experiments, signed in). A run can be logged from any machine: a GPU launched with Kovara Run, a notebook, a laptop.
Log a run
- Create a key with the
experiments:writescope at Account → Developer and give it to your script asKOVARA_API_KEY. Put the tracker next to your script. It is one Python file that needs only the standard library (Python 3.8 or newer):
curl -fsSLO https://www.kovaragrid.com/sdk/kovara_track.pyThe same file is
kovara.trackin thekovarapackage (packages/kovara-cli).- Log from your training loop:
import kovara_track as kt run = kt.init(project="llm-tune", config={"lr": 2e-4, "model": "Qwen/Qwen2.5-7B-Instruct"}) for step in range(steps): loss = train_step() run.log({"train/loss": loss}, step=step) run.summary({"eval_accuracy": 0.84}) run.finish()
Tracking never stops a training script. Without a key, or when Kovara cannot be reached, the script carries on and one line on stderr says so. While Kovara is unreachable, points are kept in memory (up to 50,000) and sent again.
| Call | What it does |
|---|---|
kt.init(project, name, config, job_type, tags, notes, price_per_hour_usd, run_key, …) | Starts a run and returns it. Every argument is optional. |
run.log({"name": value, …}, step=n) | Logs metric values for one step. Give step (for example the global training step) so a continued run lines up; without it the tracker counts calls. |
run.update_config({…}) | Adds or changes settings after the start. |
run.summary({…}) | Results to keep with the run: a final score, the address of the weights. |
run.finish(status) | Sends what is left and ends the run: finished (default), failed or stopped. |
with kt.init(…) as run: | Ends the run for you; an exception ends it as failed. |
kt.hf_callback(run) | A Hugging Face Trainer callback: Trainer(…, callbacks=[kt.hf_callback(run)]). |
A script that ends without calling finish() is ended for it; a script that crashes is marked failed. A run whose machine disappears stays "running"; after ten minutes without a signal the console shows No signal and lets you close it.
| Variable | Meaning |
|---|---|
KOVARA_API_KEY | The key (kvk_…, scope experiments:write). |
KOVARA_PROJECT, KOVARA_RUN_NAME, KOVARA_RUN_KEY, KOVARA_PRICE_PER_HOUR_USD | Defaults for the arguments of the same name. |
KOVARA_API_URL | Kovara's address (default https://www.kovaragrid.com). Must be https://; plain http:// only for localhost. |
What is sent
The settings and metrics you log, and the machine as the script sees it: GPU model, count and memory from nvidia-smi, the SkyPilot cluster, cloud and region when the script runs under SkyPilot, host name, Python and library versions, and the git commit of the working directory. Every 30 seconds it also logs GPU utilisation, GPU memory in use, GPU power and RAM in use under system/. Pass environment=False or system_metrics=False to kt.init() to turn either off. No file contents are read, and no environment variables other than the SKYPILOT_ and KOVARA_ ones named here.
The key is sent only to the configured address, and redirects are never followed.
Spot interruptions
A run started with a run_key it has used before continues instead of starting a second run. Under SkyPilot the tracker uses SkyPilot's task id as the run_key, so a managed job that is interrupted and recovered stays one run. Log with an explicit step so the curve continues where it stopped. In a notebook each kt.init() is a run of its own. The time between the interruption and the recovery is part of the run's duration.
Fine-tune a model
Launch GPU → Fine-tune a model starts a LoRA fine-tuning job on a GPU in your own cloud account. Choose the GPU, the base model and the data, and press Start fine-tuning.
| Field | Meaning |
|---|---|
| Base model | Any causal language model on Hugging Face (default Qwen/Qwen2.5-0.5B-Instruct). Larger models need a GPU with more memory. |
| Training data | A dataset on Hugging Face (default databricks/databricks-dolly-15k), or a .jsonl, .json, .csv or .parquet file in your bucket, such as /data/train.jsonl. |
| Bucket with your data | Optional. Attached at /data. Your SkyPilot server must be able to read it. |
| Save the adapter to | Optional. A bucket attached at /output. With it the adapter outlives the machine, an interrupted job continues from its last epoch, and the result is registered under Models & datasets. Without it the adapter stays on the machine's disk under ~/outputs. |
| More options | Epochs, learning rate, batch size, longest example in tokens, LoRA rank, how many examples to use, the name in the registry, and a Hugging Face token for gated models. |
A row of the data may be chat messages (messages, or conversations in the ShareGPT layout), an instruction with an answer (instruction, prompt or question with response, output, completion or answer, and optionally context or input), or plain text. Rows in another shape are skipped and counted in the run's settings.
What runs is written into the task file and shown under Templates & images → What it runs: pip install of torch, transformers, datasets, peft and accelerate, Kovara's tracker (downloaded, then refused unless its SHA-256 is the one in the task), and the job itself (finetune.py, in full). The job trains in bf16 where the GPU supports it, keeps 10% of the data (at most 200 examples) for evaluation when there are 200 or more, logs loss and GPU load to Experiments, and saves the adapter. The machine deletes itself after the idle minutes you set (10 by default) once the job has ended.
With Track in Experiments ticked, Kovara creates an API key for the machine (scope experiments:write, 30 days, named Launch <name> under Account → Developer) and it is sent to your SkyPilot server as a secret with the job, as is a Hugging Face token if you enter one. Kovara does not store either: the activity log keeps the task file without their values. Secrets need SkyPilot 0.10 or newer on your server. If the launch is refused, the key is revoked.
Notebook
Launch GPU → Notebook starts Jupyter Lab on a GPU, listening on localhost only; open it through the SSH tunnel shown after the launch. The home folder holds Kovara's tracker and kovara_start.ipynb, which checks the GPU and logs a first run. With Track in Experiments ticked the key is already set, so kt.init() in any notebook on that machine starts a run. Each kt.init() in a notebook is a run of its own.
Cost
A run's cost is its hourly price times its duration. The price comes from one of two places, and the console says which:
- The price your script stated:
kt.init(price_per_hour_usd=2.49). - Your SkyPilot server's price for the cluster: when the run reports a SkyPilot cluster and that server is connected under Cloud accounts, Kovara reads the cluster's cost and time up from the server's cost report (the report Usage & costs shows) and keeps the hourly rate with the run.
A run with neither shows no cost. Kovara never fills in a price from market data. The estimate covers the machine for the time of the run; your provider's invoice is the real figure.
On a run page and in a comparison, charts can be drawn against step, time or cost (money spent by then). Cost to reach a target in a comparison shows the first moment each run's metric got to a value, and what the run had cost by then.
Ask Kova about your runs
Kova, the assistant, can read your own runs when you are signed in. It reads them the way the Experiments pages do, as you, and changes nothing.
- "What did my last run cost?", "Which run was cheapest?": your runs with status, duration, machine, cost and the last value of each result. A run without a known price has no cost, and Kova says so.
- "Was that the right GPU?", "Make this cheaper": Kova reviews one run against Kovara's current public offerings and reports only what the evidence supports:
- The same GPU for less. The same model with the same memory, on a machine with the same number of GPUs, listed at least 5% below what the run paid per GPU-hour. Only on-demand listings checked within 30 days and not graded D are compared, the cheapest per provider. The saving shown is that price difference times this run's GPU-hours.
- A smaller GPU to try. Only when the run never used more than 60% of its GPU's memory; the smaller GPU must hold the peak plus a quarter. A smaller GPU is usually slower and Kovara has not measured your workload on it, so this comes with prices and no saving.
- Low utilisation. When the GPU was busy under half the time, the bottleneck is elsewhere (data loading, batch size), and Kova says what to change first.
What could not be checked is listed: a run with no GPU readings, a GPU Kovara has no record for, a run with no known price. Listed prices are public listings with the date they were checked, not quotes.
Models and datasets
The registry keeps names, version numbers and addresses. The files stay in your own storage: Kovara stores no weights and no data, and removing an entry never deletes files.
data = run.use_dataset("support-tickets", "production") # the address of that version; the run is linked to it
…
run.log_model("support-adapter", "s3://my-bucket/adapters/run-17", aliases=["candidate"])| Call | What it does |
|---|---|
run.log_model(name, uri, aliases, size_bytes, digest, metadata, notes, description) | Registers a version of a model this run produced and returns it. run.log_dataset(…) is the same for a dataset. |
run.use_model(name, version) | Records that this run used a version and returns its address. version is a number (3 or "v3"), an alias, or "latest" (default). run.use_dataset(…) is the same for a dataset. |
- Versions are numbered 1, 2, 3 … per entry. A number is never reused, so
v2keeps meaning the same files. - An alias such as
productionnames one version of an entry; giving it to a new version moves it.latestandv1,v2… are reserved. - An address is
scheme://…(s3://,gs://,az://,r2://,hf://,https://…). One with a user name or password in it, or with a query string (a signed link), is refused, so no credential ends up in the registry. - Registering the same address from the same run again returns the existing version.
Each version shows the run that produced it and the runs that used it, and each run page lists what it produced and used. Versions can also be registered, given aliases and removed by hand under Experiments → Models & datasets.
HTTP API
All calls take Authorization: Bearer kvk_…. Writes need the experiments:write scope, reads experiments:read.
| Method | Path | What it does |
|---|---|---|
POST | /api/v1/experiments/runs | Start a run. Body: project, name, job_type, config, environment, tags, notes, cluster_name, price_per_hour_usd, run_key (all optional). 201 {run, url}; 200 when a run was continued. |
POST | /api/v1/experiments/runs/{id}/log | Body: points (at most 2,000 of {key, step, value, t?}), config, summary. An empty body is a heartbeat. Logging a key and step again replaces the value. |
POST | /api/v1/experiments/runs/{id}/finish | Body: status (finished, failed, stopped), summary. |
GET | /api/v1/experiments/runs?project=&limit= | Your runs, newest first. |
GET | /api/v1/experiments/runs/{id}?max_points= | One run with its metric series, each thinned to at most max_points (20 to 1000, default 400). A point is [step, mean, min, max, count, seconds since the start]. |
POST | /api/v1/experiments/registry/versions | Register a version. Body: kind (model, dataset), name, uri, and optionally run_id, size_bytes, digest, metadata, aliases, notes, description. |
POST | /api/v1/experiments/registry/uses | Body: run_id, kind, name, version. Records the use and returns the version with its address. |
GET | /api/v1/experiments/registry/items?kind= | Your models and datasets. |
GET | /api/v1/experiments/registry/items/{kind}/{name} | One entry with every version, its producer and its users. |
Errors are { "error": "…" } with 401 (key), 403 (scope), 404 (not yours or not there), 422 (invalid; nothing stored), 429 (a limit) or 503.
Limits
| Runs per account | 5,000; 300 new runs per hour |
| Metric points | 200,000 per run; 5,000,000 per account; 2,000 per request |
| Metrics per run | 200 names (letters, digits and _ . / : -, at most 80 characters) |
| Settings, results, machine | 64 KB, 32 KB and 16 KB of JSON |
| Registry | 2,000 entries per account; 1,000 versions per entry; 10 aliases per version |
A long series is thinned for display: consecutive steps are read in equal groups, and each point is the mean of its group with the smallest and largest value kept, so a spike stays visible.
Data and access
Runs and registry entries belong to the account that owns the key. The tables have row-level security with no policies; every read and write goes through database functions that check the signed-in user or the key (hashed, unrevoked, unexpired, with the scope). Deleting a run removes its metric points; registry versions it produced stay, without the link. Migrations: 20261112140000_experiments.sql, 20261112150000_experiments_registry.sql.