Model Quantization from First Principles

Model quantization is the process of replacing high-precision numerical representations, commonly $32$-bit floating point, with lower-precision formats such as $16$-bit floating point, $8$-bit integers, or $4$-bit integers to reduce memory, bandwidth, and compute cost during neural network inference. At its core, quantization is not a trick specific to neural networks; it is an application of numerical approximation: map a continuous or high-resolution set of real values into a finite set of discrete values, then compute with those discrete values as efficiently as possible.

A neural network layer computes transformations such as:

$$y = Wx + b$$

where $W$ is a matrix of weights, $x$ is an activation, and $b$ is a bias term. In full precision, $W$, $x$, and $b$ are often stored as floating-point numbers. Quantization asks: can we approximate $W$ and $x$ using integers while keeping the output $y$ close enough for the task?

The most common production scheme is affine quantization, which maps a real value $r$ to an integer value $q$ through a scale $s$ and zero point $z$:

$$q = \operatorname{round}\left(\frac{r}{s}\right) + z$$

and reconstructs an approximate real value as:

$$\hat{r} = s(q - z)$$

The central idea is therefore simple: choose $s$ and $z$ so that important real values fit into a small integer range, such as $[-128,127]$ for signed $8$-bit integers or $[0,255]$ for unsigned $8$-bit integers.

Footnotes

1. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference - Foundational paper describing integer-only neural network inference and quantization-aware training methods. ↩ ↩²

2. Quantization in Digital Signal Processing - Background on mapping continuous or high-resolution values to a finite set of discrete values. ↩

3. TensorFlow Lite 8-bit Quantization Specification - Specification describing scales, zero points, signed integer ranges, per-axis quantization, and operator constraints. ↩ ↩²

Why Quantization Works

Neural networks are often robust to small numerical perturbations because many learned representations are distributed across many parameters and activations. This means that replacing a value $r$ with a nearby approximation $\hat{r}$ may not significantly change the final prediction if the induced quantization error remains small relative to the model’s margins and layer sensitivities.

For a uniform quantizer, the real line is divided into equal-width intervals. If values are clipped to a representable interval $[r_{\min}, r_{\max}]$ and encoded with $N$ integer levels, the step size is approximately:

$$s = \frac{r_{\max} - r_{\min}}{N - 1}$$

For $8$-bit unsigned quantization, $N = 256$; for signed $8$-bit quantization, there are also $256$ distinct integer codes. Smaller $s$ means finer resolution but a narrower representable range. Larger $s$ covers a wider range but increases rounding error. Quantization is therefore a trade-off between clipping error and rounding error.

A typical scalar quantization pipeline is:

$$r \rightarrow q = \operatorname{clip}\left(\operatorname{round}\left(\frac{r}{s}\right) + z, q_{\min}, q_{\max}\right) \rightarrow \hat{r} = s(q - z)$$

The clipping operation ensures that $q$ remains inside the valid integer range, such as $q_{\min}=-128$ and $q_{\max}=127$ for signed $8$-bit tensors.

Footnotes

1. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference - Foundational paper describing integer-only neural network inference and quantization-aware training methods. ↩ ↩²

2. TensorFlow Lite 8-bit Quantization Specification - Specification describing scales, zero points, signed integer ranges, per-axis quantization, and operator constraints. ↩ ↩²

3. Quantization in Digital Signal Processing - Background on mapping continuous or high-resolution values to a finite set of discrete values. ↩

The Core Equation: Real Matrix Multiplication with Integer Arithmetic

Consider a linear layer:

$$y = Wx$$

If both $W$ and $x$ are quantized, then:

$$W \approx s_W(q_W - z_W)$$ $$x \approx s_x(q_x - z_x)$$

Substituting into the matrix multiplication gives:

$$y \approx s_W s_x (q_W - z_W)(q_x - z_x)$$

The expensive part can now be expressed as integer multiply-accumulate operations, often accumulating into $32$-bit integers before rescaling to the next layer’s output format. This is the foundation of integer-only inference, a major reason quantized inference can be faster on supported CPUs, DSPs, NPUs, and mobile accelerators.

For a dot product of length $n$:

$$y = \sum_{i=1}^{n} w_i x_i$$

the quantized approximation is:

$$y \approx s_W s_x \sum_{i=1}^{n} (q_{w_i} - z_W)(q_{x_i} - z_x)$$

In practice, optimized libraries avoid unnecessary per-element subtraction by algebraically expanding the terms and precomputing sums where possible.

Footnotes

1. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference - Foundational paper describing integer-only neural network inference and quantization-aware training methods. ↩ ↩² ↩³

Symmetric vs Asymmetric Quantization

Two common forms of uniform quantization are symmetric quantization and asymmetric quantization.

In symmetric quantization, the real range is centered around zero, and the zero point is usually fixed to $0$ for signed integer formats:

$$\hat{r} = s q$$

This is especially common for weights because trained weights are often roughly centered around zero, and using $z=0$ simplifies arithmetic.

In asymmetric quantization, the range need not be centered around zero:

$$\hat{r} = s(q-z)$$

This is useful for activations after functions such as ReLU, where values may be mostly nonnegative and the distribution is shifted away from zero.

Per-channel quantization often improves accuracy for weights because different output channels can have very different ranges. TensorFlow Lite’s quantization specification, for example, supports per-axis quantization for certain weight tensors and distinguishes activation and weight constraints.

Footnotes

1. TensorFlow Lite 8-bit Quantization Specification - Specification describing scales, zero points, signed integer ranges, per-axis quantization, and operator constraints. ↩ ↩² ↩³ ↩⁴ ↩⁵

Calibration: Estimating Activation Ranges

Weights are known after training, but activations depend on input data. Calibration runs sample inputs through the model to observe activation ranges before finalizing quantization parameters.

For post-training static quantization, a representative dataset is passed through the model while observers record tensor statistics such as minimum and maximum values or histograms. These statistics determine scales and zero points for activation tensors.

A naive min-max calibration strategy uses:

$$r_{\min} = \min(x)$$ $$r_{\max} = \max(x)$$

over observed calibration activations. However, outliers can make $r_{\max}-r_{\min}$ very large, increasing $s$ and reducing precision for the majority of values. More advanced calibration strategies may use percentiles or histogram-based criteria to balance clipping error and rounding error.

Footnotes

1. PyTorch Quantization Documentation - Framework documentation covering observers, calibration, dynamic quantization, static quantization, and quantization-aware training. ↩ ↩² ↩³ ↩⁴

Dynamic, Static, and Quantization-Aware Training

There are three major deployment patterns: dynamic quantization, static quantization, and quantization-aware training.

Dynamic quantization quantizes weights in advance but computes activation scales at runtime. This is commonly useful for models dominated by linear layers, such as recurrent networks and transformer components, because it reduces weight memory while avoiding calibration of every activation tensor.

Static quantization quantizes both weights and activations before deployment using calibration. It can provide stronger speedups on hardware with integer kernels because both operands in major matrix multiplications can be low precision.

Quantization-aware training, or QAT, simulates quantization during training by inserting fake-quantization operations into the forward pass while maintaining trainable parameters in floating point. This allows the model to adapt to quantization noise and often improves accuracy when low-bit post-training quantization is too lossy.

Footnotes

1. PyTorch Quantization Documentation - Framework documentation covering observers, calibration, dynamic quantization, static quantization, and quantization-aware training. ↩ ↩² ↩³

2. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference - Foundational paper describing integer-only neural network inference and quantization-aware training methods. ↩ ↩²

Quantization Error: A First-Principles View

Quantization replaces $r$ with $\hat{r}$, so the scalar error is:

$$e = r - \hat{r}$$

For an ideal uniform rounding quantizer without clipping, the maximum absolute rounding error is approximately:

$$|e| \leq \frac{s}{2}$$

This bound explains why smaller scale values improve precision. However, reducing $s$ while keeping the same bit width shrinks the representable range and can increase clipping. Thus total error can be understood as:

$$\text{total error} = \text{rounding error} + \text{clipping error}$$

In neural networks, error compounds through layers. If one layer’s output is badly quantized, the next layer receives a distorted input distribution. This is why activation quantization is often harder than weight-only quantization: activations vary by input, layer, batch, and deployment distribution.

A useful layerwise approximation compares floating-point and quantized outputs:

$$\Delta y = y_{\text{float}} - y_{\text{quant}}$$

Large $\|\Delta y\|$ at a layer indicates that the layer may need higher precision, per-channel quantization, better calibration, clipping adjustment, or QAT.

Footnotes

1. PyTorch Quantization Documentation - Framework documentation covering observers, calibration, dynamic quantization, static quantization, and quantization-aware training. ↩ ↩²

Weight-Only Quantization and Large Language Models

For large language models, weight-only quantization is widely used because model parameters consume large amounts of memory, and reducing weight precision can substantially reduce memory footprint. For example, moving from $16$-bit weights to $4$-bit weights can reduce raw weight storage by approximately $4\times$, ignoring scale metadata and packing overhead.

In transformer inference, memory bandwidth is often a major bottleneck, especially when serving large models with many parameters. Weight-only quantization reduces the amount of weight data read from memory while often keeping activations and accumulations in higher precision for stability.

Modern LLM quantization methods frequently use grouped quantization. Instead of one scale for an entire tensor or one scale per channel, a scale may be assigned to a small group of weights:

$$W_g \approx s_g q_g$$

where $g$ indexes a group. Smaller groups improve local accuracy but increase metadata overhead because more scales must be stored.

Some advanced methods, such as GPTQ, use second-order approximations to quantize weights while compensating for quantization error layer by layer. Other approaches, such as activation-aware weight quantization, identify salient weights based on activation statistics and protect them during quantization.

Footnotes

1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers - Research paper on accurate post-training weight quantization for large transformer models. ↩ ↩² ↩³ ↩⁴

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration - Research paper describing activation-aware protection of salient weights during low-bit LLM quantization. ↩

Practical Design Choices

A quantization strategy is a set of engineering decisions, not a single formula. The main choices are:

1. Bit width: $8$-bit quantization is often a strong accuracy-efficiency trade-off for many neural networks, while $4$-bit quantization is more aggressive and usually needs more careful methods.
2. Granularity: Per-tensor quantization has low overhead; per-channel or grouped quantization improves accuracy when distributions differ across channels or groups.
3. Symmetry: Symmetric quantization simplifies arithmetic; asymmetric quantization better handles shifted distributions.
4. Calibration method: Min-max calibration is simple; histogram or percentile methods may better handle outliers.
5. Training involvement: Post-training quantization is simpler; QAT can recover accuracy by exposing the model to simulated quantization noise during optimization.
6. Hardware target: The best quantization scheme depends on what the deployment runtime accelerates efficiently.

Footnotes

1. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference - Foundational paper describing integer-only neural network inference and quantization-aware training methods. ↩

2. TensorFlow Lite 8-bit Quantization Specification - Specification describing scales, zero points, signed integer ranges, per-axis quantization, and operator constraints. ↩ ↩² ↩³

3. PyTorch Quantization Documentation - Framework documentation covering observers, calibration, dynamic quantization, static quantization, and quantization-aware training. ↩ ↩²

Evaluation Checklist

A rigorous quantization evaluation should compare the quantized model against the floating-point baseline across multiple dimensions:

The most important principle is to evaluate on the deployment path, not only in a development notebook. Quantization changes numerical formats, but real-world performance depends on graph conversion, operator fusion, memory layout, kernel availability, and hardware execution.

Footnotes

1. TensorFlow Lite 8-bit Quantization Specification - Specification describing scales, zero points, signed integer ranges, per-axis quantization, and operator constraints. ↩