30gb+, yeah. You can't get by streaming the model's parameters: NVMe isn't fast enough. Consumer GPUs and Apple Silicon processors boast memory bandwidths in the hundreds of gigabytes per second.
To a first order approximation, LLMs are first memory limited, and second bandwidth limited and if a model fits in VRAM, we can estimate single batch throughput as Memory Bandwidth / (Active Parameters * Parameter Size).
An 8-bit quantized Llama 2 70B conveniently uses 70GiB of VRAM (and then some, let's ignore that.) The M3 Max with 96GiB of VRAM and 300GiB/s bandwidth would have a peak throughput around 4.2 tokens per second.
Quantized models trade reduced quality for lower VRAM requirements and may also offer higher throughput with optimized kernels, largely as a consequence of transfering less data from VRAM into the GPU die for each parameter.
Mixture of Expert models reduce active parameters for higher throughput, but disk is still far too slow to page in layers.
<https://news.ycombinator.com/item?id=39891191>
To a first order approximation, LLMs are first memory limited, and second bandwidth limited and if a model fits in VRAM, we can estimate single batch throughput as Memory Bandwidth / (Active Parameters * Parameter Size).
An 8-bit quantized Llama 2 70B conveniently uses 70GiB of VRAM (and then some, let's ignore that.) The M3 Max with 96GiB of VRAM and 300GiB/s bandwidth would have a peak throughput around 4.2 tokens per second.
Quantized models trade reduced quality for lower VRAM requirements and may also offer higher throughput with optimized kernels, largely as a consequence of transfering less data from VRAM into the GPU die for each parameter.
Mixture of Expert models reduce active parameters for higher throughput, but disk is still far too slow to page in layers.
<https://news.ycombinator.com/item?id=39891191>
sincerely,