← All tutorials

Spectral Subtraction for Denoising

Estimate the noise, then remove it.

On this page

1 Noise adds in power 2 Learn the noise in the pauses 3 Subtract, and keep the noisy phase 4 Why the noisy phase works 5 Subtraction is a gain 6 Musical noise 7 In practice 8 When subtraction goes wrong 9 References

Spectral subtraction removes steady background noise, such as a fan next to a phone call, from speech. It learns the noise power in each frequency bin during pauses, takes that power away from every frame of the noisy spectrogram, keeps each bin’s noisy phase, and turns the result back into sound. It acts once per STFT frame of the clip, and its two settings decide how much extra to subtract and how low a bin may fall.

noisy audioSTFT framesnoise fromthe pausessubtractper binkeep thenoisy angleinverse STFT
Figure 1. The method in order, 1 to 6.

1 Noise adds in power

The STFT (The STFT and Spectrograms) cuts audio into short frames and gives one complex value per frame and frequency bin. Call the speech S, the noise D (for disturbance) and the noisy value Y; noise that simply adds to the speech adds in every bin, Y = S + D, because the recording chain is linear. The fan is stationary (its statistics stay put) while the speech comes and goes.

speech winsevennoise onlyspeechnoisenoisy, the sum
Figure 2. Three bins. Speech |S|² on noise |D|²; dashed, noisy |Y|², about their sum.
In power, the two parts add up on average.
|Y|² ≈ |S|² + |D|²
Squaring Y = S + D gives |S|² + |D|² plus a cross term. Speech and fan are unrelated, so the cross term averages to zero, and ≈ means on average. It builds on independence, and Section 3 takes the noise part away.

2 Learn the noise in the pauses

Average |Y|² bin by bin over frames with no speech, such as the first fraction of a second or frames a voice activity detector marks quiet. That is Welch’s average from Power Spectral Density done per bin, and it gives the noise PSD PD for every bin.

Audio editors do the same (Anushiravani 2016). Adobe Audition asks you to select a stretch with noise only, captures it as a noise profile, and then lets you set how much to remove. With no clean pause, take a low percentile of each bin over the whole recording instead, the 10th to 20th, since speech lifts only some frames and the lowest ones are still noise. It fails when fewer than a fifth of the frames are noise only.

0 10 dB mean 12.8 dB median 11.2 dB 20th percentile 0 dB 12 frames, power in one bin
Figure 3. One bin by a fan, illustrative. Speech lifts the mean; the percentile stays on noise.

A toy recording, a tone that starts at 0.5 s in steady noise, and its noise PSD from the frames before the tone.

import numpy as np
import librosa

SR, N_FFT, HOP = 8000, 256, 128
TONE_HZ, ONSET_S, NOISE_RMS = 440, 0.5, 0.3
t = np.arange(2 * SR) / SR
voice = np.sin(2 * np.pi * TONE_HZ * t) * (t > ONSET_S)
rng = np.random.default_rng(0)
y = voice + NOISE_RMS * rng.standard_normal(t.size)
Y = librosa.stft(y, n_fft=N_FFT, hop_length=HOP)    # (129, frames)
P_D = np.mean(np.abs(Y[:, 1:11]) ** 2, axis=1, keepdims=True)
  1. 1A toy recording, a 440 Hz tone from 0.5 s in steady noise.
  2. 2The STFT frames (marker 2 in Figure 1).
  3. 3The noise PSD PD, the mean power over frames 1 to 10 (marker 3 in Figure 1).

3 Subtract, and keep the noisy phase

Boll (1979) subtracted the average noise magnitude from each bin’s magnitude and set negative results to zero. Berouti et al. (1979) worked in power, subtracted a little more than the estimate (α) and stopped at a small floor (β). Lim and Oppenheim (1979) reviewed these rules beside the Wiener filter.

One bin of a speech frame, noise estimate PD = 1, α = 2 and β = 0.01. A noisy power of 5 keeps 5 − 2 = 3. A noisy power of 1.5 would fall to −0.5, so it is lifted to the floor, 0.01. Both keep their noisy angle (illustrative).

The subtraction rule for one bin of one frame.
|Ŝ|² = |Y|² − α PD
|Y|² is the noisy power in the bin and PD the noise estimate of Section 2. Ŝ is the cleaned bin; the hat marks an estimate. α, the over-subtraction, takes away a little more than the average noise when it is above 1, so noise that pokes above its average goes too.
|Ŝ|² ← max( |Ŝ|² , β PD )
The arrow means replace. β, the floor, is a small part of the noise (0 < β ≪ 1), so no bin drops below zero or to an empty hole. α = 1 with β = 0 is plain power subtraction.
∠Ŝ = ∠Y
∠ is the angle of the bin. Only the length changes, and the cleaned bin keeps the noisy angle; Section 4 shows why that works. The inverse STFT then rebuilds the audio. It builds on the noise estimate of Section 2, and Figure 4 steps through it on one frame.

The cards of Sections 1 to 3, one step at a time on one frame.
|Y|² ≈ |S|² + |D|²
The power sum of the card in Section 1.
PD = meanpauses |Y|²
The noise estimate of Section 2.
|Ŝ|² = |Y|² − α PD
Line 1 of the subtraction card in Section 3.
|Ŝ|² ← max( |Ŝ|² , β PD )
Line 2 of the subtraction card in Section 3, the floor.
∠Ŝ = ∠Y
Line 3 of the subtraction card in Section 3. The inverse STFT then rebuilds the audio.

Figure 4. Frame at 0.34 s, α = 2, β = 0.01. 12Tones at 500 and 1,500 Hz.

4 Why the noisy phase works

Every bin is a length and an angle (Fourier Transform and the FFT); subtraction changes only the length. Wang and Lim (1982) reported that, at moderate signal-to-noise ratio (SNR, speech power over noise power), the noisy phase serves about as well as the clean one, and estimating the clean phase is much harder.

Y noisyS cleancleanedsame angle, shorter
Figure 5. One bin as an arrow. 1Clean S. 2Noisy Y. 3Cleaned, Y’s angle, shorter.

5 Subtraction is a gain

Every subtraction rule can be rewritten as a gain, one number per bin and frame. With α = 2 and β = 0.01, a bin holding 4 times the noise power keeps about 0.71 of its amplitude, and a bin at the noise level keeps only the floor, 0.1.

The same rule as a gain on each bin.
γ = |Y|²PD
γ compares this frame's bin power with the noise estimate. It is called the a posteriori SNR because it is measured after the frame arrives, from the noisy power itself. At γ = 1 the bin holds no more than the noise, and the larger γ is, the more likely speech is there.
G = √max( 1 − αγ , βγ )
The subtraction rule with its floor, divided by |Y|², is the share of power to keep, and the square root turns that power share into an amplitude gain. As γ grows, αγ shrinks and the gain rises toward 1; once γ falls below α + β the floor takes over.
Ŝ = G · Y
A gain per bin and frame, multiplied in; the noisy angle rides along. It is at most 1, except where the floor lifts a bin that sat below β PD (γ under β). It builds on the subtraction rule of Section 3, and the Wiener filter picks a better G.
−10−50510152000.51γ in dB (bin power over the noise estimate)gain G
Figure 6. Gain against γ. 1α = 1. 2α = 2, deeper. 3Floor β = 0.01, never 0.

A gain per frequency is a filter’s magnitude response (How FIR and IIR Filters Shape Magnitude and Phase), only this one is redrawn every frame.

6 Musical noise

The noise estimate is an average, but each frame’s noise jumps around it by about its own size (the jitter of a periodogram, one frame’s power spectrum with no averaging, on Power Spectral Density). In five bins of one pause frame with noise estimate 1 and powers [0.6, 1.4, 2.5, 0.9, 3.1], α = 1 leaves 3 survivors, α = 2 leaves 2 and α = 3 leaves 1 (illustrative).

What survives subtraction in a pause.
|Ŝ|² = |D|² − α PD
In a pause the noisy power is only this frame’s noise |D|². Wherever it pokes above α PD the difference stays positive and survives as a short isolated peak; a larger α leaves fewer of them. It builds on the subtraction rule of Section 3, and Figure 7 shows the survivors.

Scattered over time and frequency these survivors sound like faint warbling tones, known as musical noise. Post-filters, run after the subtraction, are a further fix.

Figure 7. 1Noisy. 2Cleaned. Larger α thins tones; a floor hides specks left.

12340%20%40%60%over-subtraction αshare of binsnoise left in the pausesweak speech removed
Figure 8. From Figure 7, no floor. 1Noise left, 36 to 2%. 2Speech lost, 41 to 56%.

7 In practice

StepHow often, on what dataState it keepsCost
Noise estimateonce from leading silence, or again in each pausethe noise PSD, one value per binone mean per bin
Subtractiononce per frame and bin, on the noisy STFT frameonly the noise PSDa subtract, a max and a square root per bin, plus the STFT

Rates, state and cost come from the sources in References.

The subtraction rule on the same toy recording, with α = 2 and β = 0.01.

import numpy as np
import librosa

SR, N_FFT, HOP = 8000, 256, 128
t = np.arange(2 * SR) / SR
voice = np.sin(2 * np.pi * 440 * t) * (t > 0.5)      # 440 Hz tone from 0.5 s
y = voice + 0.3 * np.random.default_rng(0).standard_normal(t.size)
Y = librosa.stft(y, n_fft=N_FFT, hop_length=HOP)     # (129, frames)
P_D = np.mean(np.abs(Y[:, 1:11]) ** 2, axis=1, keepdims=True)

ALPHA, BETA = 2.0, 0.01
power = np.maximum(np.abs(Y) ** 2 - ALPHA * P_D, BETA * P_D)
S_hat = np.sqrt(power) * np.exp(1j * np.angle(Y))
s_hat = librosa.istft(S_hat, hop_length=HOP, length=len(y))
  1. 1The toy recording and its noise estimate, as in the snippet in Section 2.
  2. 2The two settings, α and β.
  3. 3Subtracts α PD and keeps at least the floor β PD (the subtraction card in Section 3, lines 1 and 2).
  4. 4Keeps the noisy angle (the subtraction card in Section 3, line 3).
  5. 5Rebuilds the audio by inverse STFT (marker 6 in Figure 1).
SettingDefaultSource
Frames256 samples, hop 128, at 8 kHzBoll (1979)
Knob, where it entersRaise itLower itDefault
Over-subtraction α, the multiplier on PDLess musical noise, more speech removedα = 1 is plain subtraction; below 1 noise is left inAbout 4.75 at −5 dB SNR falling to 1 at 20 dB, per frame (Berouti et al. 1979)
Floor β, the lowest power kept, β PDHides musical noise under a steady hissDeeper gaps, more musical noise; β = 0 is no floor0.005 to 0.1 (Berouti et al. 1979)
Noise averaging, the frames averaged into PDSteadier, but needs longer silence and tracks change slowly; past the pause it folds in speechJumpier, more musical noise; one frame is a single periodogramThe pause before speech, the first 0.5 s in the toy code of Section 2

For noise that changes over time, such as traffic, learned noise templates do better (Non-negative Matrix Factorization, section 7.1).

8 When subtraction goes wrong

What you seeLikely causeWhat to do
A door slam or passing car stays inNoise that changes is not in the estimateUpdate the estimate in every pause, or track it
Speech is removedSpeech leaked into the frames used as noiseCheck the quiet frames, use a voice activity detector
Clicks or level jumps at frame edgesAnalysis and synthesis STFT settings differSame FFT size, hop and window both ways; pass length to istft
Good SNR, bad soundSNR does not measure musical noiseJudge by spectrograms and listening too
Some bands stay noisy while others lose speechThe noise is colored differently across bandsCompute the SNR, the noise profile and the subtraction per band

Every rule on this page multiplies each bin by a gain. The Wiener Filter finds the best one.

9 References