What is Model Quantization?

Large Language Models are massive. A 70-billion parameter model stored in 16-bit precision requires roughly 140GB of VRAM-more than most consumer GPUs can handle. Quantization is the process of reducing the precision of these weights (e.g., from 16-bit to 4-bit) to make models smaller and faster.

Precision vs. Range

Computers typically store numbers using Floating Point (FP) representations, which can represent a huge range of very tiny and very large numbers with high precision. However, neural networks are surprisingly robust. They don't always need to know if a weight is 0.824159; knowing it's "about 0.8" is often enough.

Quantization takes these continuous values and maps them onto a discrete grid. The challenge is finding a mapping that preserves the most important information-the "outlier" weights that have a high impact on the model's behavior.

Post-Training Quantization (PTQ)

The most common method is PTQ, where a model is trained normally in high precision and then converted afterward. This involves:

1. Calibration: Passing a small amount of data through the model to see the typical range of weights and activations.
2. Scaling: Finding a constant value that maps the FP16 range to the target integer range (like -127 to 127 for 8-bit).
3. Rounding: Converting the values to the nearest integer.

Why it Matters

Without quantization, the "AI Revolution" would be restricted to massive data centers. Techniques like bitsandbytes (4-bit) and GGUF allow researchers and hobbyists to run state-of-the-art models on laptops. It is the bridge between theoretical research and local, private execution.