Hardware
Memory bandwidth
How much data a GPU can read from or write to its memory each second, usually measured in terabytes per second.
Why it matters
Generating text token by token is often limited by memory bandwidth rather than raw compute, so it strongly affects inference speed.
Want a deeper answer?
Ask Kova to explain Memory bandwidth in the context of your workload. Meet Kova →