Splitting non-negative data into parts that only add.
Non-negative matrix factorization (NMF) writes a non-negative data matrix, such as a spectrogram, as a few non-negative templates times how strongly each is used, so the parts can only add.
NMF iterates on one whole matrix. In supervised use the templates are learned beforehand from clean examples of each sound (Section 7.1), and only the activations are fitted for each new recording.
A spectrogram is a grid whose rows are frequencies and whose columns are short time slices, and each cell says how loud that frequency is at that moment. Each column is a mix of a few spectral templates, each turned up or down over time. NMF is matrix factorization with non-negative factors, the same idea recommender systems use with user and movie factors.
Take an illustrative spectrogram with 4 frequency bins and 3 frames, with a low note in frame 1, a high note in frame 2 and both in frame 3, as when two piano keys are played one after the other and then together. This is how NMF tells which note sounds when in a music recording, and which part of a noisy recording is speech (Section 7).
| Case | Data V | Factors W and H |
|---|---|---|
| This toy, 4 by 3, 2 templates | 12 | 8 + 6 = 14 |
| Illustrative 10 s clip at 16 kHz, 257 bins by 624 frames, 25 templates | 160,368 | 6,425 + 15,600 = 22,025 |
How many numbers the data and the factors store.
On the toy the factors are bigger than the data; on a real spectrogram they are about 7 times smaller, which forces the templates to be shared.
W and H are unknown; Section 3 scores a guess.
With no subtraction allowed, blocks cannot cancel each other, so each block must look like a real piece of the data (Lee and Seung 1999). PCA is computed with the SVD, whose blocks may be negative. Figure 3 takes the SVD of V as it is, without PCA's subtraction of each bin's mean. Vector quantization goes the other way; like k-means, each frame takes exactly one block, winner takes all. NMF sits between, a few blocks per frame that only add.
Both fits are exact with 2 blocks, but the SVD builds the high note from block 1 minus block 2, a low-minus-high contrast, while NMF's blocks are the two notes themselves.
Before W and H are known, start from a rough guess, two templates that each lean toward one note, with every activation 1. Each cell of the rebuild is then 1.5, one template giving 1 and the other 0.5, and the two costs below score how far that is from V.
| Start guess | Value, low bin to high |
|---|---|
| Template 1 in W | 1, 1, 0.5, 0.5 |
| Template 2 in W | 0.5, 0.5, 1, 1 |
| Every activation in H | 1 |
| Rebuild WH | 1.5 in every cell |
The start guess, scored here and updated in Section 4.
Section 4 lowers these costs while keeping every entry non-negative.
Each update multiplies every entry by a ratio, what the data asks for over what the model already gives. From the start guess of Section 3, one update of H under squared error runs like this; WT is W turned on its side, its transpose.
| Quantity | Value |
|---|---|
| WTV, the data side | 3, 1.5, 4.51.5, 3, 4.5 |
| WTWH, the model side | 4.5 in every cell |
| Half the gradient, model side minus data side | 1.5, 3, 03, 1.5, 0 |
| New H, H minus that row times H over 4.5 | 0.667, 0.333, 10.333, 0.667, 1 |
| Squared error | 11.0, then 5.56 after H, 3.68 after W |
The first update of H from the start guess, under squared error.
Deriving the gradient row from the squared error of Section 3 shows why this step size turns a gradient step into a multiplication.
Figure 5. Steps 0 to 3 are one fit; step 4 is a separate stuck run.
Each KL line is also a gradient step with a step size per entry, chosen as in the gradient card so that the step becomes a ratio. The authors prove the cost never goes up and state that it stops changing at a stationary point, where the gradient is zero so the update no longer moves (the same authors, 2001, Theorems 1 and 2). An entry stuck at 0 also stops changing, and a global minimum is not guaranteed, as the separate run in step 4 of Figure 5 shows. Alternating the two lines of either card is the whole algorithm.
Section 5 chooses how many templates to fit.
Too few templates blur sounds together; too many split one sound or copy noise. On the two-note example, rank 1 leaves a squared error of 5.0, and ranks 2 and 3 bring it to about 0. In its two-burst example the thesis assumes the number of sources is known, and points to a way to estimate it (p. 45).
Take an illustrative recording with three sounds, a steady hum that stops partway, speech-band bursts like syllables, and a keyboard click every 12 frames. A fit starts from random positive templates and activations, scaled to the data's loudness and fixed so every visit shows the same fit. One round is steps 1 and 2 of Figure 5, the H line and then the W line, and no round can raise the error.
round 300, squared error about 0
Figure 6. 1 Rounds find sounds; 2 too few share, too many split.
At rank 3 each sound gets its own template; the click template dips at the speech bins and the speech template fills that dip back in. At rank 2 the hum and the click share one template, so the clicks come back faint; at rank 4 the hum splits across two blue templates and the error does not drop further. Rank 1 settles by round 3 at a squared error of about 30, since one template can only average the three sounds.
x is any mono recording. This is the unsupervised version, one fit on one recording with no templates learned beforehand. Passing V with frequency bins as rows makes scikit-learn's W the templates and H the activations, the names on this page; scikit-learn's own docs warn that the literature often swaps them.
import numpy as np
import scipy.signal as ss
import sklearn.decomposition as sd
_, _, Z = ss.stft(x, nperseg=512)
V = np.abs(Z)
nmf = sd.NMF(3, solver="mu")
nmf.set_params(beta_loss=1)
W = nmf.fit_transform(V)
H = nmf.components_Z and its magnitude V, the matrix of Section 1.beta_loss=1 is the KL cost of Section 3 and needs solver="mu". The default start here is "nndsvda", built from the SVD with its zero entries filled by the data average, so it is the same on every run.W is 257 bins by 3 and H is 3 by the number of frames.The next block runs the squared-error updates of Section 4 on the two-note spectrogram of Figure 2.
import numpy as np
Wt = np.c_[[2,1,0,0], [0,0,1,2]]
V = Wt @ [[1, 0, 1], [0, 1, 1]]
rng = np.random.default_rng(0)
W = rng.random((4, 2)) + 0.1
H = rng.random((2, 3)) + 0.1
e = 1e-9
for _ in range(200):
H *= W.T@V / (W.T@W@H + e)
W *= V@H.T / (W@H@H.T + e)V from the two notes and their activations, as in Figure 2.+ 0.1 keeps every entry above 0, since a 0 never changes under multiplication (Section 8).e, which avoids dividing by 0, the loop and the activation update.It returns the two notes up to scale, so each template can come back multiplied by some number, with its activation row divided by the same number.
Separation with fixed templates comes first, then the way back to sound with the mixture phase, templates that last several frames, and the thesis's room matching.
Learn speech templates from clean speech and noise templates from noise alone. On a noisy recording, keep both sets fixed and fit only the activations; the speech templates times their activations rebuild the speech (the thesis, section 4.7.1, eqs 4.27 to 4.29).
Each STFT cell is a complex number. Its size is the loudness kept in V, its angle says where the wave is in its cycle, and NMF throws the angle away. Say NMF rebuilds one noisy cell as speech 3 plus noise 1.
Speech 3 plus noise 1 rebuild a size of 4, while the real cell has size 5. Using the speech magnitude 3 directly would ignore that miss; dividing by the rebuild and scaling the real cell corrects the size and borrows the angle (the thesis notes this compensates for “misestimation in the magnitude spectrogram”, p. 49).
Continuing the first block of Section 6, the next block keeps template k's share of every cell, puts it on the complex Z and inverts. Section 7.1's supervised version fixes W before fitting; this one just picks a template.
k = 0
Vk = np.outer(W[:, k], H[k])
mask = Vk / (W @ H + 1e-9)
Sk = mask * Z
_, xk = ss.istft(Sk, nperseg=512)k's part and divides by the full rebuild, the mask of the first card line.Z, so the mixture phase is kept, and inverts, the second card line.Plain NMF treats every frame alone. Sounds and rooms have time patterns, so Section 7.3 lets templates last several frames.
A sound is a short movie, not one spectrum. In convolutive NMF (Smaragdis 2004) each template lasts a few frames and slides along time, the same sliding weighted sum as an FIR filter.
Plain NMF would need two separate templates, (2, 0) and (1, 1), and would lose the fact that one always follows the other.
Learned from clean speech, such templates also serve the supervised separation of Section 7.1 (Smaragdis 2007).
The thesis's goal is example-based editing, making a recording sound drier or making it sound like the room of an example recording. A room smears each sound with a decaying tail of echoes, the dry sound convolved with the room's response. In the thesis model speech is still built from the same dry templates, and each template's activation row is convolved with its own decaying tail.
The thesis learned the speech templates once offline (25 templates, 500 iterations), then decomposed each 10-second recording (section 5.13). Plain NMF and its mask undo a blend of parts in one recording, and convolutive NMF extends that to blends over time, the counterpart of the blend across sensors that Independent Component Analysis undoes.
The table lists the settings that matter in scikit-learn 1.9.1 and in the thesis, what each one does, and its default.
| Setting | Raise it | Lower it | Extremes | Default |
|---|---|---|---|---|
n_components, the rank K | finer parts and lower error; past the true number, sounds split or noise gets a template | sounds merge into shared templates (Figure 6) | a rank as large as the smaller side of V rebuilds it exactly and teaches nothing | "auto", taken from W or H when given; the thesis used 25 speech templates (p. 76) |
max_iter | lower cost, slower | unconverged, blurry templates | multiplicative updates slow down near the end | 200; 500 in the thesis (p. 76) |
init | A choice, not a scale. "nndsvda" is the SVD-based start of Section 6; "random" depends on the seed. | identical columns never separate (stuck at 5.0, step 4 of Figure 5) | None, meaning "nndsvda" when n_components is at most the smaller side of the data | |
beta_loss, not the kernel exponent β | A choice, not a scale. "frobenius" or 2 is squared error; "kullback-leibler" or 1 is the KL cost and needs solver="mu". | a choice; no extreme | "frobenius"; KL in the thesis (eq 4.25) | |
alpha_W, alpha_H, l1_ratio | sparser, more part-like templates or activations; too high zeros things out | plain NMF | 0 turns the penalty off | 0, "same", 0 |
| Thesis sparsity powers on the activations and the kernel (Section 7.4) | nearer 1, a weaker push | a stronger push, which the thesis describes as sparser | a power of 1 turns the push off | activations 0.9 to 0.98, kernel 0.8 to 0.9, applied each iteration (p. 72) |
| Size of the starting decay factor λ (thesis, Section 7.4) | toward 1.5, the starting kernel dies away faster | toward 0.9, the starting kernel dies away slower | not given in the thesis | λ between −1.5 and −0.9 per frequency band (eq 5.33, p. 72) |
| Kernel length (thesis) | longer room tails can be captured, with more unknowns | tail cut short, leftover reverb | too short cannot hold a hall | 45, 65 and 85 frames for a bedroom, a conference room and a living room (p. 76) |
| Kernel exponent β (thesis) | shorter tail, drier sound | longer tail, more reverb | 1 leaves the room unchanged | fitted per component between 0.5 and 2 (eq 5.36, p. 74) |
| Cause | Symptom | Fix |
|---|---|---|
| identical starting templates | stuck at a rank 1 answer, error 5.0 in the two-note example | a random or "nndsvda" start |
| a zero entry in W or H | it stays zero forever under multiplication | start every entry above 0, as + 0.1 does in the second block of Section 6 |
| wrong rank | too low merges sounds, too high splits them or models noise | compare a few ranks on held-out audio and listen |
| the reused mixture phase | traces of the original after separation or dereverberation; “some traces of the original reverberation can often persist” (Anushiravani and Smaragdis) | accept it or use a phase-aware method |
Common NMF failures, what they look like and the fix.
Feed NMF a magnitude STFT, as the thesis does, not power; power squares every cell, so the loudest bins would rule the fit even more. Scale the templates to peak 1 before comparing runs, since a template and its activation row can trade any scale. The cost should fall fast and then flatten; a curve still falling at max_iter needs more iterations, and a curve flat at a high value points to a bad start or too low a rank.
Next in the menu is t-SNE, which maps data to two dimensions for plotting rather than splitting it into parts.