← All tutorials

The STFT and Spectrograms

A spectrum for every moment.

On this page

1 One spectrum for the whole clip hides when things happen 2 Frames, windows and the DFT 3 Time resolution and frequency resolution 4 Windows and leakage 5 Going back with overlap-add 6 In practice 7 When a spectrogram looks wrong 8 References

The short-time Fourier transform (STFT) cuts a signal into short overlapping frames, fades the edges of each frame, and takes the FFT of each one. It runs once per frame, a new frame every 10 ms in a typical speech system, so one second is 100 FFTs. The output is a spectrogram, with time across, frequency up and brightness for strength, and the frame length trades sharp timing against sharp pitch.

1 One spectrum for the whole clip hides when things happen

Sound changes over time, but one spectrum of a whole clip adds every moment together.

rising chirp 0 1 s 4k 2k 0 falling chirp 0 1 s whole-clip spectrum, same for both 0 4 kHz
Figure 1. 1A rising tone and 2the same clip backward. 3One whole-clip spectrum.

2 Frames, windows and the DFT

Cut the signal into short frames of L samples, each starting R samples (the hop) after the last. Fade each frame’s edges with a window w[n], take its DFT, and stand the result up as one column. Figure 2 does it to a tone that jumps from 500 to 1,500 Hz.

One frame of the STFT.
xm[n] = x[n + mR] w[n]
Frame m starts mR samples in, R being the hop. The window w fades its L samples at the edges, so the cut adds no clicks.
X[m,k] = Σn xm[n] e−j2πkn/N
n runs over the frame, 0 to L − 1, k is the bin and N ≥ L the DFT size. Frame m gives column m of the spectrogram. It builds on the DFT, and Section 5 runs it backward.

Figure 2. 1A frame of L samples. 2Windowed. 3Its DFT, one column.

Frames usually overlap by a half or three quarters. The DFT size N can exceed L by padding with zeros, which adds bins but no detail.

3 Time resolution and frequency resolution

Dennis Gabor showed in 1946 that no signal is sharp in both time and frequency at once. In an STFT the window length L sets the balance. At 8 kHz, a 64-sample window spans 8 ms and blurs frequency over 125 Hz; a 1,024-sample one spans 128 ms and blurs it over only 7.8 Hz.

What one window length buys and what it costs.
Δf = fsL
The frequency resolution of a window of L samples, and the bin spacing when the DFT size N equals L. At fs = 8 kHz, L = 64 gives 125 Hz and L = 1,024 gives 7.8 Hz. Two pitches share one band when they are closer than about Δf with no window, or about 2Δf with the smooth Hann window of Section 4: the 50 Hz pair of Figure 4 merges at 32 ms, where 2Δf is 62.5 Hz, and splits at 64 ms, where it is 31 Hz. Zero-padding the frame to N above L puts bins fsN apart but leaves Δf alone, because the frame still holds L real samples.
Tw = Lfs
The window spans Tw seconds, and everything inside it lands in the same column, so a click spreads over that span: 8 ms for 64 samples, 128 ms for 1,024. Each column is drawn at its frame’s center, so the long-window image of Figure 3 starts and ends 64 ms, half a window, in. Multiply the two lines and Δf times Tw is 1: raising L sharpens frequency and blurs time by the same factor.
Δt = Rfs
Columns sit Δt seconds apart, one hop. A smaller hop R draws more columns per second but cannot undo the spread of line 2, because each column still sums a whole window.
short window, L = 64 (8 ms)01k2k Hz00.51time, slong window, L = 1,024 (128 ms)01k2k Hz00.51time, s
Figure 3. L = 64, 1thin click, 2thick tone. L = 1,024, 3wide click, 4thin tone.

Figure 4. 1The window. 2Clicks 40 ms apart. 3Tones at 1,000 and 1,050 Hz.

4 Windows and leakage

The DFT treats a frame as one period of a signal that repeats forever. A frame cut with no fade, w[n] = 1, uses the rectangular window; the Hann window fades both ends to zero.

rectangularHann windowseam between copies
Figure 5. 10.5 cycles per frame. 1Rectangular copies jump. 2Hann copies meet smoothly.

A jump contains every frequency, so a tone that does not fit a whole number of cycles spills energy into far bins. This is leakage. A window’s spectrum is a central main lobe, whose width sets how close two tones can be, plus smaller side lobes, whose height sets how far a loud tone leaks. Heights are in decibels (dB) below the peak; every 20 dB down is ten times smaller in amplitude.

Three windows of length L, each a short sum of cosines.
θ = 2πnL
θ is the angle at sample n; it grows by one full turn, 2π, across the L samples of the frame.
w[n] = 12 (1 − cos θ)
Hann is one raised cosine, zero at the frame’s start, one in the middle and back toward zero at the end, so the copies meet without a jump. Its highest side lobe is 31.5 dB down and the lobes keep falling 18 dB per octave, that is, each time the distance from the peak doubles, but the main lobe is 4 bins wide, twice the rectangular window’s 2.
w[n] = 0.54 − 0.46 cos θ
Hamming puts the same cosine on a step, so the ends sit at 0.08 instead of 0. The weights 0.54 and 0.46 are picked so the first side lobe nearly cancels, 42.4 dB down here, but because the ends do not reach zero the far lobes fall only 6 dB per octave.
h = 0.42 − 0.5 cos θ
Blackman starts from a Hann-like cosine h on a smaller constant, 0.42 instead of 0.5.
w[n] = h + 0.08 cos 2θ
A second cosine at twice the speed brings both ends back to zero and flattens them further. The highest side lobe drops to 58 dB down and the main lobe widens to 6 bins, so close tones blur more than with Hann. All three build on the frame of Section 2, and Section 5 needs their shifted copies to add up flat.
−13.3 dB2 binsRectangular0−50−100main lobeside lobes−31.5 dB4 binsHann0−50−100−42.4 dB4 binsHamming0−50−100−58.1 dB6 binsBlackman0−50−100bins from the peak, −10 to 10
Figure 6. L = 64. 1Top side lobe (dashed). 2Main lobe (bar). Lower side lobes, wider main lobe.

Figure 7. 1The window in orange. 2Bars over the dash are within 50 dB.

Window3 dB widthHighest side lobePick it when
Rectangular0.89 bins−13 dBthe tone sits exactly on a bin, or nothing loud sits nearby
Hann1.44 bins−32 dBin doubt; the default in SciPy and librosa
Hamming1.30 bins−43 dBleakage into the nearest bins matters more than into far ones
Blackman1.68 bins−58 dBa weak tone must stay visible next to a strong one

Values from Harris (1978), Table 1, rounded; the last column is common practice.

5 Going back with overlap-add

Denoising edits a spectrogram and then needs the sound back.

inverse DFT of each column, slid to mRtheir sum over the signal
Figure 8. 1Inverse DFTs at mR. 2Overlap-add. 3A flat window sum gives the signal.
sum of the windows hop R = L/2
Figure 9. 1Hann windows sum flat 2at hop R = L/2. 3As one fades out, the next fades in.

A 4-point Hann window, [0, 0.5, 1, 0.5], slid by 2 samples adds to 1 at every sample, 0 + 1 and 0.5 + 0.5.

Inverse-transform every column, slide each frame back into place and add.
Σm w[n − mR] = C
Plain overlap-add, which only adds the frames, gives the signal back when the shifted windows add up to a flat line, a constant C, so every sample gets the same total weight. This is COLA, constant overlap-add. Hann meets it only when the hop splits L into a whole number of steps, R = L2, L3, L4 and so on; at 25 ms windows every 10 ms, 2.5 steps, it does not.
Σm w2[n − mR] ≠ 0
scipy.signal.istft and librosa.istft window each inverse frame again, add, and divide every sample by this sum of squared windows, so they need only that it is never zero. This is NOLA, nonzero overlap-add. Hann with hop 300 of 400 fails line 1 but passes line 2 and still rebuilds the signal exactly.

A run of ones checks line 1 (Anushiravani 2016): send a constant through the STFT and plain overlap-add, and if it comes back flat, the window and hop meet COLA. Windowing again on the way back without that division, weighted overlap-add (WOLA), needs the squared windows to add to a constant instead; the square root of a Hann window does, summing to 1 at hop L/2 and to 2 at hop L/4.

The round trip in SciPy at hop 300, where the Hann windows no longer sum flat, with the largest error printed at the end.

import numpy as np
from scipy import signal

FS, L, R = 16000, 400, 300
print(signal.check_COLA('hann', L, L - R), signal.check_NOLA('hann', L, L - R))   # False True
w = signal.get_window('hann', L)
starts = range(0, FS - L + 1, R)                # one window every hop
ones_back = sum(np.pad(w, (m, FS - L - m)) for m in starts)
print(np.ptp(ones_back[L:-L]))                  # 0.71, top to bottom
x = np.random.default_rng(0).standard_normal(FS)
_, _, Z = signal.stft(x, fs=FS, window='hann', nperseg=L, noverlap=L - R)
_, x_back = signal.istft(Z, fs=FS, window='hann', nperseg=L, noverlap=L - R)
print(np.max(np.abs(x - x_back[:len(x)])))         # 3.8e-15, near machine precision
  1. 1Sets a 400-sample Hann window with hop 300 and prints False and True: the windows do not sum flat (eq card in Section 5, line 1), but their squares never sum to zero (line 2). The ones test agrees: plain overlap-add of a constant comes back with a ripple of 0.71 from top to bottom.
  2. 2Makes one second of noise and takes its STFT (eq card in Section 2, lines 1 and 2).
  3. 3istft windows each inverse frame again, adds the frames and divides by the summed squared windows; the largest error, 3.8e-15, is near machine precision.

This is how Spectral Subtraction for Denoising rebuilds audio after cleaning the spectrogram.

6 In practice

Speech systems usually take 25 ms frames every 10 ms, because speech is close to stationary, its spectrum nearly steady, over about 20 to 30 ms. At 16 kHz that is 400 samples with a hop of 160, and the DFT size is the next power of two, 512. The speech recognizer Conformer reads features on exactly such 10 ms frames, and CTC Loss, one loss used to train such recognizers, lines the transcript up with them.

The speech settings in librosa, window L = 400 and hop R = 160. One second of samples goes in, and the complex STFT comes out with 257 bins by 101 frames.

import numpy as np
import librosa

SR, N_FFT, L, R = 16000, 512, 400, 160
x = np.random.default_rng(0).standard_normal(SR)
X = librosa.stft(x, n_fft=N_FFT, hop_length=R, win_length=L)
power = np.abs(X) ** 2
print(power.shape)                  # (257, 101)
  1. 1Sets the speech settings, 25 ms windows every 10 ms at 16 kHz, and one second of noise.
  2. 2Takes the STFT with L = 400 and R = 160 (eq card in Section 2, lines 1 and 2).
  3. 3Squares each bin’s magnitude into its power and prints its shape, 257 bins by 101 frames.
StepHow often, on what dataState it keepsCost
STFTonce per frame, every hop, on one windowed framenone; frames are independentone FFT per frame
Overlap-addonce per frame, on one frame’s inverse FFTthe overlapping tail of earlier framesone inverse FFT and one add per sample
SettingDefaultSource
Kaldi frames25 ms window, 10 ms shifttorchaudio.compliance.kaldi
librosa.stftn_fft 2048, hop n_fft//4, Hann, centered frameslibrosa 0.11 documentation
scipy.signal.stftnperseg 256, overlap nperseg//2, periodic Hann window, which divides by L rather than L − 1 so shifted copies sum flatscipy.signal.stft
Knob, where it entersRaise itLower itDefault
Window length L, the frame in the STFT cardFiner frequency detail, blurrier timingSharper timing, blurrier frequency25 ms (torchaudio kaldi.mfcc)
Hop R, the shift from frame m to m + 1Fewer frames and less cost. Plain overlap-add needs a hop that splits L evenly; istft works at any hop (Section 5)More frames, smoother tracks, more cost10 ms (torchaudio kaldi.mfcc)
Window type w[n], the fade in the STFT cardSmoother tapers (Hann, then Blackman) lower leakage but widen each peakThe rectangular window keeps peaks narrowest but leaks into every binHann (scipy.signal.stft)
FFT size N, at least LZero padding gives a finer grid, not finer resolutionBelow L the frame would be cutWindow rounded up to a power of two (torchaudio kaldi.mfcc)

librosa documents the inverse at librosa.istft; SciPy marks scipy.signal.stft as legacy and recommends ShortTimeFFT.

6.1 Gotchas

7 When a spectrogram looks wrong

What you seeLikely causeWhat to do
A pure tone smears over many binsNo window, so the frame edges jumpUse a Hann window (Section 4)
The rebuilt signal ripples in loudnessHand-written overlap-add with windows that do not sum flat at that hopDivide by the summed squared windows as istft does, or use Hann at 50% overlap
The first frames look quieterlibrosa pads the start with zeros to center framesExpect it, or pass center=False

How is the power of noise spread over frequency? That is Power Spectral Density.

8 References