Size the GPUs, runtime and cost of an AI workload before you spend.

Memory: weights, KV cache and what is left

Before speed, RA-BIT asks whether the model fits. Weights cost their parameter count times the bytes of the precision. For serving, the KV cache grows with every token in flight, so the memory left after the weights decides how many sequences one replica can decode together.

weights   = params × bytes            (BF16 2 B · FP8 1 B · INT4 ≈ 0.52 B at group 128)
KV/token  = 2 · L · n_kv · d_head · bytes
budget    = HBM · 0.9 · gpus − 1.5 GB · gpus − weights − workspace
sequences = budget ÷ KV per sequence  (whole paged blocks)

For training, mixed-precision Adam holds 16 bytes per trainable parameter, sharded by tensor and pipeline parallelism and then by ZeRO stage 1, 2 or 3 over data parallelism. Activations follow Korthikanti et al. (arXiv 2205.05198), with or without recomputation. LoRA keeps the frozen base and trains only the adapters.

Speed: prefill is compute, decode is memory

Serving has two phases with different physics. Prefill processes the whole prompt in one pass: many FLOPs per weight read, so tensor cores set the pace. Decode produces one token per sequence per iteration and must read every weight and the KV cache each time, so at small batch the memory bus sets the pace. RA-BIT times each scheduler iteration as the slower of the two limits.

FLOPs = 2 · N_active · T + 4 · L · (H · d_head) · A
Bytes = weights read + KV read + KV written
t     = max( FLOPs ÷ (tp · peak · mfu_max · T ⁄ (T + 64)),
             Bytes ÷ (tp · BW · η_bw) )
        + 2 · L · AllReduce(T · h · 2 B, tp) + step overhead

Parallelism and topology

Splitting a model across GPUs adds collectives. RA-BIT uses an alpha-beta model per fabric, picks the cheaper of ring and tree latency, and overlaps communication with compute only partly. Topology is per part: NVSwitch parts reach any peer at full per-GPU bandwidth; MI300X is a full mesh of seven Infinity Fabric links, so a group of p GPUs uses p − 1 of them; PCIe parts go over the host bus.

Serving simulation under load

Closed-form averages hide queues. For inference RA-BIT runs a deterministic, iteration-level discrete-event simulation of one replica under continuous batching, seeded so the same inputs always give the same output.

• Poisson or closed-loop arrivals; prompt and output lengths fixed, uniform or lognormal.
• First-come first-served with a token budget, chunked prefill or prefill priority, a maximum batch.
• KV admission with preempt-and-recompute or full reservation; timeouts; requests that can never fit are rejected.
• TTFT, ITL, end-to-end and queue p50 / p95 / p99 with sample counts: p95 needs at least 20 samples and p99 at least 100, otherwise it is withheld.
• Throughput, utilisation, KV occupancy, prefill time share, rejection rate and a saturation flag.

Training runtime and goodput

Training is sized by FLOPs per token, then by the fastest layout that fits memory: tensor × pipeline × data × expert parallelism × micro-batch × ZeRO stage, each evaluated for memory and time. The deadline search covers node-aligned counts up to 32,768 GPUs. FP8 speeds the compute by 1.3 to 1.5 times, never the 2× peak ratio.

Search, Pareto front and market prices

The optimizer enumerates configurations, prunes those that cannot fit or cannot meet the latency target even at batch 1, screens the rest in closed form, simulates up to 16 survivors, sizes replicas from simulated p95s, and keeps the Pareto front on cost, end-to-end p95 and capacity. The recommendation is the cheapest priced configuration that meets the target and the hard constraints, with a structured explanation: why this GPU, why this count, the bottleneck, assumptions, risks, and cheaper or faster alternatives.

Uncertainty and calibration

RA-BIT does not attach a confidence score to a single number. It runs the model at the low and high bound of each assumption and combines the deviations, which gives a heuristic range. When comparable measured runs exist, it compares each run with its own prediction and corrects the model by the mean log-ratio.

range:  low = base · exp(−√Σ dᵢ↓²),  high = base · exp(√Σ dᵢ↑²)
rᵢ     = ln(measuredᵢ ⁄ analyticalᵢ),   bias = mean rᵢ
value  = analytical · exp(bias)
n ≥ 3:  interval = value · exp(± t₀.₉₅,ₙ₋₁ · sd · √(1 + 1/n))
n < 3:  heuristic range ×0.7 … ×1.3

The next experiment, and diagnostics

Some uncertainties change the decision and some do not. RA-BIT moves each uncertain input (decode throughput per accelerator, prompt length, arrival rate) to its low and high bound, re-runs the optimizer, and checks whether the choice of accelerator, layout or precision flips. Experiments are ranked by that flip chance per GPU-hour of measuring, with the kovara CLI command to run.

Once a saved analysis receives telemetry, ten explicit rules with stated thresholds look for a cause. A rule fires only when every metric it needs is present; with no telemetry there is no diagnosis.