One OpenAI-compatible endpoint in front of the inference providers you already use. A route is a model alias, such as llama-70b, that maps to an ordered list of targets: a provider endpoint and that provider's model name. Kovara sends each call to the first target and moves to the next one on rate limits, outages and timeouts. Every call is logged with tokens, latency and cost at your own contracted prices.
Set it up at Deployments → Inference router (/deployments/inference, signed in).
Call it
Create a key with the inference:invoke scope at Account → Developer, then point any OpenAI client at Kovara:
from openai import OpenAI
client = OpenAI(base_url="https://www.kovaragrid.com/api/v1/inference", api_key="kvk_…")
reply = client.chat.completions.create(model="llama-70b", messages=[{"role": "user", "content": "Hello"}])| Method | Path | What it does |
|---|---|---|
POST | /api/v1/inference/chat/completions | Chat completions through a route. model is the route alias. stream: true is supported. |
POST | /api/v1/inference/completions | Text completions (the older prompt format), same rules. stream: true is supported. |
POST | /api/v1/inference/embeddings | Embeddings. input is a string or 1 to 2048 items. No streaming. |
GET | /api/v1/inference/models | Your route aliases, in the OpenAI list format. |
Kovara replaces model with each target's provider model and passes every other field through unchanged: messages, tools, response formats, sampling parameters and so on. Only n = 1 (and best_of = 1) is supported. Request bodies can be up to 4 MB. A route can serve any of the three operations; the provider path follows the operation (/chat/completions, /completions, /embeddings under the endpoint's base URL).
Responses carry X-Kovara-Route, X-Kovara-Endpoint (URL-encoded name of the endpoint that answered) and X-Kovara-Attempts. Errors use the OpenAI error shape ({"error": {"message", "type"}}), so OpenAI SDKs show the message.
Failover rules
- Moves to the next target: network errors, no response headers within the route's timeout (5 to 120 seconds, 60 by default), and HTTP 408, 409, 425, 429 and 5xx.
- Returned at once: any other error, for example 400 (bad request), 401 or 403 (the provider refused your key), 404 (unknown model) or 422. Trying another provider would hide a configuration or request problem.
- Streams fail over before the first byte only. Once a provider starts streaming, its stream is passed through as is.
- When every target fails, the answer is 502, or 429 if the last target was rate-limited. The message names the last endpoint and its error.
Cost and usage
Tokens come from the usage object the provider returns. For streams Kovara asks for it with stream_options.include_usage and reads it from the final event as the stream passes through. Cost is computed from the prices you enter on each target (USD per million input and output tokens, from your provider contract). Kovara does not estimate prices or tokens: with no price, or no usage report, the cost is shown as unknown and those calls are counted as "unpriced".
Embeddings report input tokens only; their output tokens are counted as zero, so a priced embeddings call costs its input tokens at the input price.
Monthly budgets
A route can have a monthly budget in USD. Once the route's cost in the current UTC month reaches it, new calls are refused with HTTP 429 and a message naming the route and the amount spent, until the month ends or the budget is raised. A budget needs both prices on every target of the route, so no call can run uncounted. The routes table shows each route's spend this month against its budget.
The console shows the last 7 days by route and day: calls, errors, failovers, tokens, cost and p50/p95 latency. It also lists the latest 50 calls with the failover path each one took.
Endpoints
Any OpenAI-compatible base URL works. Give the base URL up to the version segment; Kovara appends /chat/completions. Examples:
- A hosted provider, such as
https://api.together.xyz/v1. Check the provider's documentation for the exact base URL. - Your own vLLM or SGLang server behind HTTPS, for example a service endpoint started with Kovara Run.
Security
- Provider keys are stored in Supabase Vault. The database row keeps only the Vault reference and the last four characters. Keys are decrypted only by a server-side function, and only after your Kovara key has been checked (hashed, unrevoked, unexpired,
inference:invokescope). They never reach a browser and are removed from every error message. - Each provider key is sent only as the
Authorizationheader to that endpoint's own base URL. - Calls go only to public host names over HTTPS. IP addresses, private, loopback, link-local and cloud-metadata addresses are refused at connection time, and redirects are not followed.
- Prompts and answers are not stored. The call log keeps route, endpoint, model name, status, token counts, cost, latency and a short error text.
- Each person can have up to 10 endpoints and 50 routes, and make up to 600 calls per minute.
- Removing an endpoint deletes its key from Vault. Kovara refuses to remove an endpoint while a route still uses it.
Limits
- Chat completions, text completions and embeddings are routed. Images and audio are not yet.
- A budget is checked before each call, so concurrent calls started just under it can finish above it by their own cost.
- A stream that breaks midway is not resumed on another provider.
- Prices you enter are not checked against Kovara's market data. They are your contract terms.