Memory capacity
The model weights and the KV cache must fit in VRAM. Long contexts and many concurrent users grow the cache quickly.
Loading
Inference runs continuously once you have users. It is usually limited by memory rather than raw compute, and cost depends on how many requests each GPU can serve.
Plan memory headroom, model replicas and capacity patterns before committing to a serving fleet.
The fabric carries collective operations across nodes. Ask suppliers to document topology, congestion behavior, adapters, bandwidth and whether the fabric is dedicated or shared.
Fabric · Congestion · TopologyChange the assumptions below to understand scale and estimated compute rental cost.
Illustrative eight-GPU-node topology. Hardware compatibility, fabric, storage, quoted pricing and availability must be confirmed with the supplier.
Model workload economicsA practical checklist for requesting comparable supplier proposals.
This worksheet is not a provider specification. No availability, performance or pricing is implied beyond values you enter.
Low-latency, always-on serving for real users.
The model weights and the KV cache must fit in VRAM. Long contexts and many concurrent users grow the cache quickly.
Generating text token by token is often bound by how fast memory can be read, so bandwidth strongly affects speed.
Running close to your users lowers latency, and data residency rules may fix the region for you.
Traffic rises and falls. Being able to add or remove GPUs matters more than for a fixed training run.
One to eight GPUs per model replica, chosen so weights plus cache fit comfortably. Larger-memory GPUs can serve bigger models on fewer cards.
A mix is typical: reserved or committed capacity for baseline load and on-demand for peaks. Spot suits only work that tolerates interruption.
Model fit, KV-cache growth, concurrency and geographic placement shape inference much more than peak advertised GPU FLOPS.
Weights + KV cache + runtime overhead. Long context and many simultaneous users can exhaust HBM.
Time to first token, decode rate and tail latency. A fast average can conceal slow experiences at peak load.
Replica count, batching and autoscaling. Throughput and latency move in different directions.
User geography, compliance and network path. Place compute close to demand when constraints allow.
Low-latency responses require model residency, right-sized batching and a suitable region.
GPU comparisonMeasure representative prompts and context windows.
Sizing only for the weights and forgetting the KV cache.
Choosing the cheapest region without checking latency to users.
Relying on spot capacity for user-facing traffic.
This is general guidance, not a recommendation for a specific provider or price. Check current offerings and confirm availability and terms with the provider.
See tracked offerings with their sources and dates, or ask Kova to size it for you.