Four-bit quantization made large models fit consumer GPUs

Compressing weights after training, not during it

Post-training quantization reduces the numeric precision used to store a model's weights after training finishes, rather than training a smaller model from the start. The GPTQ paper, submitted in October 2022, proposes a one-shot method that uses approximate second-order information to decide how to round each weight to three or four bits per parameter, aiming to keep the output close to the original full-precision model rather than simply rounding weights independently.

What GPTQ reported

The paper reports quantizing a 175-billion-parameter model in roughly four GPU-hours, describing the accuracy loss relative to the uncompressed baseline as negligible on the tasks it measured, and states the method more than doubled the compression gains of previously proposed one-shot quantization approaches. It also reports that a 175-billion-parameter model could, for the first time, run inference inside a single GPU once quantized, with end-to-end inference speedups of about 3.25 times on an A100 and 4.5 times on the less expensive A6000 compared to an unquantized FP16 baseline. A later paper on activation-aware quantization reported that protecting a small fraction of weights identified by their activation statistics, rather than treating all weights equally, further reduced quantization error on the benchmarks it used, presenting itself as an alternative to GPTQ's approach.

What llama.cpp added, and what quantization gives up

Separately, the llama.cpp project, whose earliest public history dates to March 2023, is a dependency-free C/C++ inference runtime rather than a quantization method; it implements its own family of integer quantization formats, from 1.5-bit through 8-bit, and targets CPU and consumer GPU hardware directly, including Apple Silicon and x86 processors, rather than only data-centre accelerators. Together, a quantization method that shrinks weights with bounded measured error and a runtime built to execute those shrunk weights on hardware people already own made running a large model outside a data centre practical. Quantization is still a lossy transform: the degradation the cited papers call negligible was measured on their own benchmark tasks at specific bit-widths, and a different task, a lower bit-width, or a different model family is a separate measurement, not a guaranteed outcome.

• What bit-width and calibration data were used, and does the reported error apply to your task or a different one?
• Was the quantized model compared against the same full-precision baseline on the same benchmark?
• Does the runtime's quantization format match the one the accuracy numbers were measured with?

The combination of a bounded-error compression method and a portable runtime is well documented; the specific loss for any one model, task and bit-width still needs its own check before relying on it.