Finding the layer that lost accuracy
On this page
A quantized model that loses accuracy almost always has one of a few causes, and each one leaves a visible symptom. Every tensor in the model runs the round trip taught on Quantization Basics, so every check here asks one question: which tensor got a bad grid?
A float here is a 32-bit number with a fractional part, the model's usual format, and int8 is an 8-bit integer, one of 256 levels. Read Quantization Basics first if the round trip is new.
FakeQuantize runs this round trip inside training during quantization-aware training (QAT).These checks run once, after the model is quantized and its accuracy has dropped, not during training (Figure 1).
The listings use one toy model, TinyCNN, a small convolutional network of the kind taught in CNN Basics, from the Post-Training Quantization and Quantization-Aware Training page (Figure 2). Imports follow that page, with tq for torch.ao.quantization. PyTorch now marks torch.ao.quantization eager and FX quantization for migration to torchao (PyTorch 2.15 quantization docs); the checks carry over.
Each row links the check that confirms it.
| Symptom | Usual cause | Fix |
|---|---|---|
| Big accuracy drop | Calibration data unlike production, or one outlier channel | Calibrate on real inputs; per‑channel weights (outliers) |
| Small drop everywhere | Unfused layers, extra int8 to float hops | Fuse conv, batch norm and ReLU (FX graph) |
| One layer misbehaves | Range far wider than its data, or clipping | Plot it against its range (plot) |
| QAT does not converge | Observers frozen too early | Freeze later (scales) |
convert_fx leaves a layer in float | No qconfig for that layer | Check the qconfig mapping (FX graph) |
prepare_fx fails | Untraceable control flow in forward | Move the branch out of forward |
The six checks of Figure 1, cheapest first, then the outlier check in 2.7.
Weight fake quants (the FakeQuantize modules of Quantization-Aware Training) keep one scale per output channel by default, so a single number would hide the bad channel. Two signs give a problem away (Figure 3). The first is a layer whose scale reads exactly 1.0 with zero point 0, often one value where one per channel was expected. That is the default PyTorch keeps when an observer never saw data, and converting such a model prints the warning must run observer before calling calculate_qparams. The second is one channel whose scale sits far above the rest.
The loop prints each weight fake quant's scale shape and its first four scales.
mods = qat_model.named_modules()
for name, mod in mods:
if name.endswith("weight_fake_quant"):
# one scale per out channel
s = mod.scale
print(name, s.shape, s[:4])An observer's range should hug the data. When the range is much wider than the data, most grid levels sit where no value falls (Figure 4).
A forward hook is a function PyTorch runs each time a layer executes, so it can record the layer's output; the histogram goes beside the observer's range (printed in Inspect the observers).
import torch
import matplotlib.pyplot as plt
outputs = {}
def record(module, inputs, out):
outputs["relu1"] = out.detach()
relu1 = model_fp32.relu1
hook = relu1.register_forward_hook(record)
with torch.no_grad():
model_fp32(calib_batch)
hook.remove()
acts = outputs["relu1"].flatten()
plt.hist(acts.numpy(), bins=100)relu1 output each time it runs.Turn one layer back to float and see whether accuracy comes back. If it does, that layer's grid is the problem.
import copy
import torch.ao.quantization as tq
import torch.ao.quantization.quantize_fx as qfx
fq = qat_model.conv1.weight_fake_quant
fq.disable_fake_quant()
# lasting fix: conv1 stays float
qmap = tq.get_default_qat_qconfig_mapping("x86")
qmap.set_module_name("conv1", None)
model = copy.deepcopy(model_fp32)
model.train()
ex = (calib_batch,)
qat_model = qfx.prepare_qat_fx(model, qmap, ex)conv1's weights and re-measure accuracy.conv1 to no qconfig and prepare the model again, so it stays float.FX (PyTorch's graph-tracing toolkit) turns the model into a list of operations. A layer with no observer row was never quantized; a quantize and dequantize pair between two layers that should run together usually means a missed fusion.
One call prints that list as a table, one row per operation and per inserted observer.
# needs tabulate
qat_model.graph.print_tabular()Each observer's recorded range sets its tensor's scale and zero point, and anything outside that range gets clamped to the grid edge, so reading the ranges shows which tensor got a bad grid. After post-training quantization the observers can be called directly; after QAT, FX adds each activation observer as a module on the top-level model, named activation_post_process_0, activation_post_process_1 and so on, and each one is a FakeQuantize that holds the observer inside it (eager-mode prepare_qat attaches one to each layer as layer.activation_post_process instead).
The PTQ listing reuses TinyCNN, model_fp32 and calib_loader from Static quantization.
In eager mode an observer can also be called directly on a tensor: a MinMaxObserver for the activations and a PerChannelMinMaxObserver for the weights, then calculate_qparams() turns each recorded range into scales and zero points.
import torch
import torch.ao.quantization as tq
# quint8 activations
act_obs = tq.MinMaxObserver()
i8 = torch.qint8
w_obs = tq.PerChannelMinMaxObserver(0, i8)
conv = model_fp32.conv1
# 8 ranges, one per channel
w_obs(conv.weight.detach())
with torch.no_grad():
for x, _ in calib_loader:
a = model_fp32.relu1(conv(x))
act_obs(a)
# 1 (scale, zero point) pair
print(act_obs.calculate_qparams())
# 8 pairs
print(w_obs.calculate_qparams())The next figure and table come from a bigger model than TinyCNN, the companion repo's fused MNIST CNN, whose layers are named conv.0, conv.2 and fc1. In our run of the companion notebook, its calibration hook printed these activation ranges.
After convert() those ranges are baked into the quantized layers. The notebook uses get_default_qconfig("fbgemm"), which like the x86 default uses 7 bits for activations (0 to 127) and the histogram observer, which trims each range a little, so each scale is a bit under the range divided by 127 (PyTorch quantization docs, Best Practices). The table lists them.
| Layer | Quantized module and its params |
|---|---|
| conv.0 | QuantizedConvReLU2d scale 0.050122, zero_point 0 |
| conv.2 | QuantizedConvReLU2d scale 0.129116, zero_point 0 |
| fc1 | QuantizedLinear qscheme torch. |
Both convolutions get zero point 0 because ReLU outputs are never negative, so float 0 sits on the first level.
After QAT the loop below reads the same numbers from each FakeQuantize.
prefix = "activation_post_process_"
for name, fq in qat_model.named_modules():
if not name.startswith(prefix):
continue
# the observer
obs = fq.activation_post_process
lo = obs.min_val.item()
hi = obs.max_val.item()
s = fq.scale.item()
z = fq.zero_point.item()
print(name, s, z, lo, hi)When the cheap checks do not point at a layer, compare every layer at once. PyTorch's Numeric Suite (torch.ao.ns._numeric_suite_fx, imported as ns) logs each layer's output in a float copy and an int8 copy on the same input and scores each pair by its SQNR (signal to quantization noise ratio). It runs on the QAT model from Quantization-Aware Training. The leading underscore means the API can move between PyTorch versions.
The first listing builds the two copies and fills their loggers with one real batch.
import copy, torch
import torch.ao.quantization as tq
import torch.ao.ns._numeric_suite_fx as ns
from torch.ao.quantization import quantize_fx
ref = copy.deepcopy(qat_model).eval()
# float copy
ref.apply(tq.disable_fake_quant)
q = copy.deepcopy(qat_model).eval()
int8 = quantize_fx.convert_fx(q)
Logger = ns.OutputLogger
ref, int8 = ns.add_loggers("fp32", ref, "int8", int8, Logger)
with torch.no_grad():
ref(calib_batch)
int8(calib_batch)The second listing prints each layer's SQNR.
import torch.ao.ns._numeric_suite_fx as ns
from torch.ao.ns.fx import utils
Logger = ns.OutputLogger
logs = ns.extract_logger_info(ref, int8, Logger, "int8")
for layer, r in logs.items():
out = r["node_output"]
fp = out["fp32"][0]["values"][0]
q8 = out["int8"][0]["values"][0]
sqnr = utils.compute_sqnr(fp, q8)
print(layer, sqnr.item())Figure 6 uses illustrative values from a five-layer CNN (conv1 to conv3, fc1 and fc2), deeper than TinyCNN, so the noise has later layers to carry it. The error carries forward, so the culprit is where SQNR first falls (Figure 6).
A range wider than its data is the usual reason one layer falls (the third row of the table in section 1). Twice the range doubles the grid step. Noise carries forward because each layer's output noise is roughly the larger of the noise it inherits and its own rounding. The first doubling costs only a couple of dB, because the layer's own rounding is still smaller than the noise it inherits. After that, each doubling costs about 6 dB, one bit. In large language models a few huge activation features, up to 20 times the others, stretch a range the same way (LLM.int8(), Dettmers et al., 2022; see Normalization and Quantization).
Figure 7 is a small toy with random weights and five dense layers named after Figure 6's, with int8 weights and 8-bit (256-level) activations. A healthy layer adds only a little rounding noise, so SQNR slips a few dB per layer, under 4 here; a fall of 6 dB or more from the layer before, a whole bit, marks the culprit. A layer kept in float with the buttons adds no rounding, so its bar shows only the noise it inherits.
Figure 7. Illustrative toy. Blame the first fall of 6 dB or more.
An outlier is one channel whose values are far larger than the rest, as in A real weight histogram; with one scale for the whole tensor, it widens the grid step for every channel (Nagel et al., 2021). A dead activation is an output stuck at zero whatever comes in, so it carries no signal.
Both show up in two numbers, the widest channel range over the median one and the number of channels that are always zero (outputs comes from the hook listing in Plot activation ranges).
# (8, 1, 3, 3)
w = model_fp32.conv1.weight.detach()
# (8,): range per channel
rng = w.abs().amax(dim=(1, 2, 3))
print(rng.max() / rng.median())
# (8,): True if a channel is 0 everywhere
# (ReLU is never negative, so max 0 means all 0)
dead = outputs["relu1"].amax(dim=(0, 2, 3)) == 0
print(dead.sum())relu1 channels that output 0 for every input in the batch; each one is a dead channel. A share of zeros over the whole layer would not show this, since a healthy ReLU already zeros many of its outputs.For an outlier channel, use per-channel weights (the default in get_default_qconfig_mapping("x86")) or a histogram observer that clips the tail. For dead activations, check the layer's input range: if everything going into a ReLU (which passes positive values and zeros the rest) is negative, the fix is upstream, not in the grid.
Once the model quantizes cleanly, the next lever is the architecture itself: neural architecture search looks for a network that is small from the start (Neural Architecture Search Basics), and Knowledge Distillation trains it from a larger teacher.