Dynamic, static and QAT workflows in PyTorch
On this page
Post-Training Quantization (PTQ) quantizes a trained model once, with no retraining, and Quantization-Aware Training (QAT) trains with the rounding on; both run the round trip from Quantization Basics on every weight and activation. Weights can be quantized any time, but an activation's range is only known once inputs flow through the model, so the three paths differ only in when that range is fixed: per batch at run time (dynamic PTQ), once on sample inputs before deployment (static PTQ), or during training (QAT) (Figure 1).
Quantize one tensor walks the round trip step by step. The listings use PyTorch's torch.ao.quantization (now marked deprecated, see Quantization in practice).
Dynamic quantization is the low-effort option: the weights are quantized once after training. Activations arrive in float32 and get quantized to int8 on the fly, per batch.
One call to quantize_dynamic takes a trained float model, returns qmodel, and swaps every nn.Linear for an int8 version (PyTorch's qint8, signed integers −128 to 127; quint8 is the unsigned 0 to 255 type).
import torch.nn as nn
import torch.ao.quantization as tq
model.eval()
layers = {nn.Linear} # default dtype qint8
qmodel = tq.quantize_dynamic(model, layers)It helps most on CPUs, for LSTMs (recurrent networks that read a sequence one step at a time; see Gated RNNs: GRU and LSTM) and Linear-heavy models such as Transformers. In our run of the companion notebook, on its small MNIST (handwritten-digit) CNN (convolutional neural network), accuracy barely moved (Figure 2).
Why not CNNs? PyTorch's dynamic quantization does not cover Conv2d, and it pays off only when loading weights dominates run time. In a CNN, reading and writing the activation maps between layers dominates, and dynamic quantization keeps those activations in float32. Static quantization keeps them int8, which is why Figure 12 sends CNNs to static PTQ.
Static quantization runs everything in int8: weights and activations. The activations' range is measured once, before deployment: calibration runs some inputs through the model (no labels needed) and records each activation's range. It suits CNNs on phones and small boards.
The listings on this page and on Debugging a Quantized Model share one toy model, TinyCNN, on 1×28×28 MNIST images, defined in the next listing.
import torch.nn as nn
class TinyCNN(nn.Module):
def __init__(self):
super().__init__()
# 1x28x28 -> 8x26x26
self.conv1 = nn.Conv2d(1, 8, 3)
self.relu1 = nn.ReLU()
self.pool = nn.MaxPool2d(2)
self.fc = nn.Linear(8 * 13 * 13, 10)
def forward(self, x):
h = self.relu1(self.conv1(x))
h = self.pool(h)
return self.fc(h.flatten(1))Calibration needs an observer, a small module PyTorch inserts after a tensor that records its min and max (or its histogram) and computes s and z from them as in Scale and zero point. In the eager-mode listing below (eager mode places stubs and fusions by hand; FX graph mode automates them), the backend is the library that runs the int8 ops ("x86" on Intel and AMD CPUs, "qnnpack" on ARM).
import copy, torch
import torch.ao.quantization as tq
fp32 = copy.deepcopy(model_fp32).eval()
model = tq.QuantWrapper(fp32)
pair = [["conv1", "relu1"]]
tq.fuse_modules(fp32, pair, inplace=True)
model.qconfig = tq.get_default_qconfig("x86")
# insert observers
tq.prepare(model, inplace=True)
# calibrate
with torch.no_grad():
for x, _ in calib_loader:
model(x)
# swap in int8 modules
tq.convert(model, inplace=True)In our run of the companion notebook, on its larger fused MNIST CNN trained for 10 epochs, accuracy held and the model shrank to a quarter of its size (Figure 3).
Fusing Conv + BatchNorm (a layer that rescales each batch of activations to a steady range) + ReLU (which zeroes negatives and passes positives) into a single op before quantizing saves memory traffic, runs faster, and removes the rounding step between the three (Figure 4). At inference the batch norm's scale and shift fold into the convolution's weights. Dynamic mode has no fusion step.
A QConfig tells PyTorch how to quantize each tensor: symmetric or asymmetric (both drawn in Scale and zero point), and per-tensor (one scale for the whole tensor) or per-channel (one scale per channel, a channel being the output of one convolution filter; Figure 5).
Per-channel gives each narrow channel its whole grid, so accuracy rises, for a few extra numbers per layer and a backend that supports it. PyTorch's x86 default uses per-channel weights and per-tensor activations (qconfig source).
In the listing, model is the TinyCNN.
import torch.ao.quantization as tq
# quint8 activations
act = tq.MinMaxObserver
# int8, one scale per output channel
wt = tq.default_per_channel_weight_observer
qc = tq.QConfig(act, wt)
model.qconfig = qcA few hundred real inputs are usually enough. The observer only knows the values calibration showed it (Figure 6): if the set is too small or does not look like production, it learns the wrong ranges and either clips real activations or wastes resolution (Figure 7).
After convert() the ranges are baked in and the int8 model is not trained further, so a bad calibration means calibrating again.
Figure 6. Calibration data flows through; each layer's min and max set its scale.
Figure 7. Illustrative batches widen the range; missing loud inputs clip the tail.
One outlier stretches a min-max range over every bin (A real weight histogram). A histogram observer applies the MSE strategy from Where the range should end (Nagel et al., 2021) to activations: it builds a histogram of what it sees during calibration, tries many clipping ranges, and keeps the one with the smallest round-trip error.
The QConfig below, as in Static quantization, swaps in a histogram observer for activations and keeps per-channel MinMax weights; model is the TinyCNN.
import torch.ao.quantization as tq
# quint8, MSE range from histogram
act = tq.HistogramObserver
# int8, one scale per output channel
wt = tq.default_per_channel_weight_observer
qc = tq.QConfig(act, wt)
model.qconfig = qcIt matters most when activations have long tails, skew, or several peaks, the cases where MinMax quietly costs accuracy.
Every quant-dequant boundary moves data from int8 to float and back, which kills performance and breaks layer fusion (Figure 8). ReLU, pooling and many elementwise ops quantize safely, so everything between the entry and exit stubs can stay int8. In eager mode the listing below places the stubs by hand, as QuantWrapper did above; FX inserts them itself.
import torch.nn as nn
import torch.ao.quantization as tq
class QuantTinyCNN(nn.Module):
def __init__(self):
super().__init__()
self.quant = tq.QuantStub()
self.body = TinyCNN()
self.dequant = tq.DeQuantStub()
def forward(self, x):
h = self.body(self.quant(x))
return self.dequant(h)FX, PyTorch's graph-tracing toolkit, turns the model into a graph, so FX quantization applies the qconfig to every nested layer, inserts the observers and stubs, and fuses layers itself.
In the FX version of the static workflow, qfx.prepare_fx traces model_fp32 with one example input and inserts observers, calibration fills them, and qfx.convert_fx returns the int8 model. To check where FX put the observers, see Read the FX graph table.
import torch
import torch.ao.quantization as tq
import torch.ao.quantization.quantize_fx as qfx
example = (torch.randn(1, 1, 28, 28),)
qmap = tq.get_default_qconfig_mapping("x86")
model = model_fp32.eval()
prep = qfx.prepare_fx(model, qmap, example)
# calibrate
with torch.no_grad():
for x, _ in calib_loader:
prep(x)
model_int8 = qfx.convert_fx(prep)When PTQ costs too much accuracy, QAT fine-tunes with the rounding switched on, so training learns around it; training with simulated quantization comes from Jacob et al. (2018). The observers become FakeQuantize modules, which let a gradient pass through the rounding (by the straight-through estimator, drawn in Figure 9).
The first listing copies the trained float model in train mode and lets qfx.prepare_qat_fx insert a FakeQuantize wherever the QAT qconfig mapping asks for one; the second trains as usual with cross-entropy, switches to eval mode and converts. Each FakeQuantize runs the round trip from Quantize one tensor in every forward pass; train_loader and optimizer are the usual ones from float training.
import copy, torch
import torch.ao.quantization as tq
import torch.ao.quantization.quantize_fx as qfx
qat_qmap = tq.get_default_qat_qconfig_mapping("x86")
example = (torch.randn(1, 1, 28, 28),)
model = copy.deepcopy(model_fp32)
model.train()
qat_model = qfx.prepare_qat_fx(model, qat_qmap, example)import torch.nn.functional as F
for x, y in train_loader:
logits = qat_model(x)
loss = F.cross_entropy(logits, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
qat_model.eval()
model_int8 = qfx.convert_fx(qat_model)QAT can learn activation clipping, push small weights out of dead zones (the band around zero that all rounds to zero), and learn quantization-friendly distributions.
Rounding and clamping have no useful gradient, so QAT uses a Straight-Through Estimator (STE): in the backward pass it pretends the rounding output equals its input.
import torch
from torch.autograd import Function
class RoundSTE(Function):
@staticmethod
def forward(ctx, x):
return torch.round(x)
@staticmethod
def backward(ctx, grad):
return grad # straight throughEach QAT step does two things, shown in the card: the forward pass computes with the rounded weights, and the backward pass copies the gradient straight back to the float weights.
A float weight can move for many steps without changing its rounded value, like an input sliding along one flat step of the staircase in Figure 9. Once it crosses a bin edge, halfway between two grid levels, the rounded weight jumps a whole level and the loss sees the change. Zero QAT steps is plain PTQ: round the trained weights once and ship them. Fewer bits mean fewer levels, so each weight lands further from its float value.
To see PTQ and QAT on real data, take handwritten 5s and 8s, 60 of each from the UCI digits set (Alpaydin and Kaynak, 1998), and squeeze each 8×8 image to two numbers with PCA, so each digit becomes a point on a flat map. A small MLP with two ReLU hidden layers of 12 units, trained in float with a squared hinge loss, which penalizes every digit that is not safely on its correct side, gets all 120 right. Its decision edge is the line on that map where it switches between calling a digit 5 and calling it 8. Accuracy here counts these same 120 digits, the ones the network trained on.
Rounding to b bits uses the card's grid, codes −k to k, so it has 2b−1 levels: one of the 2b codes is dropped so that +k and −k both exist. Each output channel gets its own scale, taken from the float weights, and biases and activations stay in float. Each QAT step trains through the rounding with the same loss and Adam, using a learning rate that falls to zero by step 200.
Rounded only (PTQ), 108 of 120 right
Figure 10. One toy run, the worst of 40 at 3 bits. QAT pulls the blue edge back.
In this run, at 3 bits, rounding alone misreads 12 digits the float network got right, the ringed ones, and two QAT steps pull the blue edge back toward the grey line and win 10 of them back, from 108 to 118 right. At 2 bits rounding misreads 30; the first three QAT steps win back only 2, and step 4 jumps to 113. From 4 bits up, rounding alone already gets all 120 right, so there is nothing left to win. QAT fits the digits, not the float edge, so where no digits sit the blue shade can stay away from the grey line, as it does at 2 bits. This run is the worst of 40 random starts at 3 bits: it loses 12 digits, while the median start loses 2. At 2 bits about 1 run in 10 stays stuck, with QAT winning back less than half.
The same trade shows up at scale. Google’s Gemma 3 QAT checkpoints (April 2025) trained about 5,000 steps against the float model’s own outputs, and afterwards 4-bit rounding raised perplexity 54% less than it did without QAT. Perplexity measures how surprised a language model is by real text, so lower is better. Meta’s quantized Llama 3.2 1B and 3B models (October 2024) used QAT with LoRA adapters for 4-bit weights. A LoRA adapter is a small pair of trainable matrices added beside a frozen weight matrix, so only a few numbers train. PTQ is cheaper and usually enough at 8 bits; QAT pays off as the bits fall.
Inside FakeQuantize the observer is usually a MovingAverageMinMaxObserver, which keeps an exponential moving average of the min and max seen in training; PTQ uses MinMaxObserver or HistogramObserver instead. prepare_qat_fx() inserts the FakeQuantize modules; convert_fx() replaces them with real quantized ops. A FakeQuantize holds its observer plus the quantize-and-read-back step, and Inspect the observers' recorded ranges prints both.
Raising c makes the range follow the latest batches faster but jump with each noisy batch; lowering it smooths the range but reacts slowly, and c = 0 freezes it. PyTorch's MovingAverageMinMaxObserver defaults to c = 0.01 (its averaging_constant, observer source).
QAT starts from the trained FP32 model and fine-tunes it with a small learning rate; PyTorch's tutorial uses SGD at 0.0001 for 8 epochs, freezing the batch-norm statistics after the fourth epoch and the observers after the fifth (static quantization tutorial).
A common schedule warms up first (Figure 11), letting the observers learn the scale and zero point while the model trains, then freezes them and keeps training on one fixed pattern of quantization noise. Freezing earlier gives more epochs on a fixed grid but a range measured on fewer batches; freezing later tracks the range longer but leaves less time to adapt.
import torch.nn.functional as F
import torch.ao.quantization as tq
import torch.ao.nn.intrinsic.qat as nniqat
for epoch in range(num_epochs):
if epoch == bn_freeze_epochs:
# freeze BN running stats
qat_model.apply(nniqat.freeze_bn_stats)
if epoch == warmup_epochs:
# freeze scale and zero point
qat_model.apply(tq.disable_observer)
for x, y in train_loader:
out = qat_model(x)
loss = F.cross_entropy(out, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()TinyCNN has no BatchNorm, so on it this line does nothing.torch.ao.quantization eager and FX quantization for migration to torchao (quantize_ for eager, prepare_pt2e and convert_pt2e for graph mode) and plans to delete it in a future release (PyTorch 2.15 quantization docs). The ideas on this page carry over unchanged; the import paths do not.The PyTorch quantization docs tie each path to a different bottleneck (Figure 12).
If accuracy still dropped, which layer lost it? Debugging a Quantized Model traces it layer by layer. Pruning and Knowledge Distillation shrink a model in other ways.