From a spectrogram to model inputs.
On this page
A speech model rarely reads raw samples or a full spectrogram. Feature extraction turns each short STFT frame, one every 10 ms or so, into a few dozen numbers that keep the smooth outline of the spectrum (its envelope), which tells sounds apart, and drop most of the pitch. The usual chain is a power spectrum, a mel filterbank that bends frequency the way ears hear, and a log, which gives the log-mel features; for MFCCs (mel-frequency cepstral coefficients) a cosine transform then orders the result from broad shape to fine detail.
Ears hear pitch in roughly equal steps of ratio, not of hertz; 100 to 200 Hz sounds like a big jump, 7,100 to 7,200 Hz like almost none. The mel scale bends frequency so that equal mel steps sound like equal pitch steps.
Slaney’s version of the scale, from his Auditory Toolbox and the default in librosa unless you pass htk=True, is a straight line below 1 kHz. Its units differ, 15 at 1 kHz instead of 1000, so Figure 2 scales it up to meet the HTK curve there and compares the shapes.
Lay triangles evenly along the mel axis, so they come out narrow at low frequencies and wide at high ones, and let each one sum the power under it.
Loudness is heard roughly on a log scale, and a louder voice or another microphone multiplies the spectrum, as any filter does (How FIR and IIR Filters Shape Magnitude and Phase).
Neighboring mel bands rise and fall together. The DCT (discrete cosine transform) rewrites the 40 log energies as amounts of cosines across the bands, ordered from slow to fast change; like principal component analysis (PCA) it packs correlated numbers into a few that matter most. Applied to the log spectrum it gives a spectrum of a spectrum, hence the name cepstrum (Fourier Transform and the FFT). Davis and Mermelstein (1980) found these coefficients the best of the representations they compared for word recognition.
The cepstrum splits a spectrum into its slow shape, the envelope, and its fine detail; that split is also one way to remove a room’s echo (Anushiravani 2016; Reverberation and Dereverberation, section 8).
Forty bands at one level give only the first coefficient, c0, the overall level; a straight tilt from 3 down to 1 across them puts almost everything else into the second, c1, one slow tilt.
Figure 5. A vowel-like frame. 1Mel bands. 2Log band energies. 3DCT coefficients.
The test sound is synthetic, harmonics under an envelope peaking near 700, 1,200 and 2,600 Hz over a breath-noise floor 20 dB down. The lowest of the 40 bands are about 45 Hz apart, so the harmonics of a 220 Hz voice show as ripple there. In Figure 6 a few coefficients rebuild only the envelope; many bring the pitch ripple back.
Figure 6. Green, 40 log band energies; red, rebuilt from the first coefficients.
One frame is a snapshot. Deltas add how each coefficient is changing, and cepstral mean normalization removes what never changes. A coefficient that rises by 0.5 every frame, as at a vowel onset, has a delta of 0.5.
librosa computes the same slope with a Savitzky-Golay filter, a sliding least-squares polynomial fit, of width 9.Simpler numbers each pick out one property of a frame. A 1 kHz tone sampled at 16 kHz changes sign at 0.125 of its neighboring sample pairs, white noise at about half.
librosa’s default), high for bright sounds and low for dull ones.Figure 7. 1Zero crossings, 2centroid, 3roll-off, 4chroma, telling the sounds apart.
One librosa call per feature on one second of noise; each returns shape (1, frames), except chroma’s 12 rows.
import numpy as np
import librosa
SR = 16000
y = np.random.default_rng(0).standard_normal(SR)
feature = librosa.feature
rms = feature.rms(y=y)
zcr = feature.zero_crossing_rate(y)
centroid = feature.spectral_centroid(y=y, sr=SR)
rolloff = feature.spectral_rolloff(y=y, sr=SR, roll_percent=0.85)
flatness = feature.spectral_flatness(y=y)
chroma = feature.chroma_stft(y=y, sr=SR) # (12, frames)feature is short for librosa.feature.| Task | Usual input |
|---|---|
| Speech recognition with a network | Log-mel, stopping after the log so the network learns the rest |
| Older speech recognition | MFCC with deltas |
| Speaker identity | MFCC with mean normalization; a Gaussian mixture can model the cloud of MFCC frames |
The Conformer (Gulati et al. 2020, Conformer) reads 80-channel log-mel frames from 25 ms windows every 10 ms, trained with SpecAugment, which blanks out random blocks of bands and of time steps. The CTC speech experiments of Graves et al. (2006, CTC Loss) used 12 MFCCs plus log energy and their first derivatives, 26 numbers per frame.
| Step | How often, on what data | State it keeps | Cost |
|---|---|---|---|
| Mel filterbank | built once at design time, on sample rate and band count | none | one-off |
| Log mel and MFCC | once per frame, on that frame’s power spectrum | none | one filterbank product, a log and a DCT |
| Deltas | once per frame, on neighboring frames (librosa uses 9) | the neighbor frames | a few adds per coefficient |
| Mean normalization | once per recording, or a running mean when streaming, on all frames | one mean per coefficient | one mean per coefficient |
Rates, state and cost come from the sources in References.
MFCCs with deltas and mean normalization in librosa, on one second of noise.
import numpy as np
import librosa
SR, N_FFT, HOP, N_MELS, N_MFCC = 16000, 400, 160, 40, 13
y = np.random.default_rng(0).standard_normal(SR)
S = np.abs(librosa.stft(y, n_fft=N_FFT, hop_length=HOP))**2
fb = librosa.filters.mel(sr=SR, n_fft=N_FFT, n_mels=N_MELS, htk=True)
mel = fb @ S
log_mel = librosa.power_to_db(mel)
mfcc = librosa.feature.mfcc(S=log_mel, n_mfcc=N_MFCC)
delta = librosa.feature.delta(mfcc)
mfcc_cmn = mfcc - mfcc.mean(axis=1, keepdims=True)| Setting | Default | Source |
|---|---|---|
| Kaldi MFCC | pre-emphasis 0.97, each sample minus 0.97 times the one before, which lifts the high frequencies | torchaudio.compliance.kaldi.mfcc |
librosa MFCC | 20 coefficients at 22,050 Hz, DCT type 2 | librosa.feature.mfcc |
Pre-emphasis is one more fixed filter in the chain after the room and the microphone, and the order of such filters does not change the result (LTI Systems and Convolution).
| Knob, where it enters | Raise it | Lower it | Default |
|---|---|---|---|
| Mel bands Nmel, the sum over ℓ in the band card | Finer frequency detail; each band is narrower and noisier | Coarser bands that blur nearby formants; one band is just the frame’s energy | 23 (torchaudio kaldi.mfcc); 40 in the tools here |
| Number of MFCCs, how many q are kept | Keeps faster ripple across bands, up to the pitch harmonics; all Nmel give back the log band energies | Only the envelope; one coefficient is just the level c0 | 13 (torchaudio kaldi.mfcc); 20 in librosa.feature.mfcc |
| Window length, the samples in each frame | Finer frequency detail, blurrier timing; one window over the whole clip gives a single frame | Sharper timing, coarser frequency detail; a window shorter than a pitch period loses the harmonics | 25 ms (torchaudio kaldi.mfcc); 2,048 samples (n_fft) in librosa |
| Hop, the step between frames | Fewer frames, coarser timing, less compute; a hop longer than the window skips samples | More frames and compute; neighbors overlap more and repeat each other | 10 ms (torchaudio kaldi.mfcc); 512 samples (hop_length) in librosa |
librosa.feature.mfcc takes keyword arguments only, so mfcc(y, sr) fails.| What you see | Likely cause | What to do |
|---|---|---|
| Features shift between training and use | Different sample rates put each triangle on different Hz | Resample to one rate first |
| Small, steady mismatch | HTK and Slaney mel mixed | Pass the same htk setting everywhere |
| Model struggles with loudness | Log forgotten, so a gain stays a multiplier | Take the log, in decibels |
| Offsets between tools | Different dB references (ref=1.0 by default vs ref=np.max) | Fix one reference |
| Great test score, poor in use | Normalization statistics taken from the test set | Use training statistics only |
| Top bands always empty | Narrowband audio upsampled to 16 kHz leaves the bands above 4 kHz empty | Use fewer bands or a lower top frequency |
How does a model turn these frames into words? Conformer walks through one speech model that reads them.