Why Audio Deepfake Detectors Struggle on New Data: What the MLN-WavLM Paper Tested
In a new study, a detector learned what "normal" sounds like from its training data, and new data moved that line. The paper, posted on arXiv on November 8, 2026, describes MLN-WavLM, an audio deepfake detector that learned only from real human speech. On a familiar benchmark it reached an 8.45% equal error rate, close to a fully supervised model at 7.86%. On a harder benchmark with new generators and compressed audio, its error rate rose to 22.12%. A detector score only means something next to the conditions it was tested in.
What did the MLN-WavLM paper try?
Supervised audio deepfake detectors are trained on examples of both real speech and known fake speech. The authors, from Czestochowa University of Technology, Sungkyunkwan University and AGH University of Krakow, point out the risk: such detectors "can learn generator-specific shortcuts that fail as synthesis systems and acoustic conditions evolve."
They tried the opposite. Their model never sees a fake during training. It learns the statistics of real speech from a frozen, pre-trained speech model (WavLM Base+) and scores any clip by how far it sits from that picture of real speech. The further away, the more likely it is synthetic.
How well did it work?
The paper reports results on two public benchmarks from the ASVspoof challenge series. Equal error rate (EER) is the point where the share of real clips wrongly flagged equals the share of fakes missed. Lower is better.
Test set | MLN-WavLM (real speech only) | Supervised WavLM-HM-Conformer |
|---|---|---|
ASVspoof 2019 LA (familiar conditions) | 8.45% EER | 7.86% EER |
ASVspoof 2021 DF (new generators, different sources, lossy compression) | 22.12% EER | 14.94% EER (the paper's supervised reference) |
On the familiar set, the real-speech-only model nearly matched the supervised one. On the 2021 deepfake set, which the authors describe as a mix of "previously unseen generators, different source material, and lossy compression," its error rate climbed to 22.12%, and the supervised reference held up better.
Why does the score move when the data changes?
The model's idea of "real" came from one set of recordings. When the real voices themselves sound different, because of new sources or compression, genuine clips start to look unusual too. The paper describes this as a weakness of its "bona-fide-calibrated operating point": the threshold set on old real speech no longer fits new real speech. It is more than a shifted threshold, though: the authors found that ranking itself is worse on the new data, so how well the detector separates real from fake also dropped.
The authors also tried to fix the threshold on the fly, re-estimating it from unlabeled incoming audio. That worked when the data stayed similar, even with up to 10% fakes mixed in. Under the 2021 shift, it led to "a total collapse of low-FPR recall," meaning that at settings tuned to flag few real clips, the detector caught almost no fakes. Their conclusion: the model "can effectively rank deepfakes," but it "does not yield a stable open-world decision rule."
What should you check before trusting a detector score?
This study is about speech, not music. Still, the same question applies to any AI audio detector, including the ones used on songs. A score is only as reliable as the match between your audio and the audio the tool was tested on.
Question to ask | Why it matters |
|---|---|
What data was the tool tested on? | MLN-WavLM's error rate rose from 8.45% to 22.12% when the test data changed. |
Which generators were included? | Unseen generators were part of the shift that hurt performance. |
Was the audio compressed or re-encoded? | Lossy compression was also part of the harder benchmark. |
Which error rate is reported? | EER, false-positive rate and detection rate are different numbers. A low EER says little about how many real tracks get flagged at your chosen threshold. |
MusicScanner, MuseBlossom's AI music analysis tool, shows an AI analysis index for a track. We treat that index as a reference signal to read alongside how and where a track was made, not as a verdict. Results like this paper's show why that distinction matters.
What this paper does not tell you
It is a preprint from one research team, posted before peer review, and every number comes from the authors' own experiments. Because it tests speech rather than songs, it says nothing direct about music detectors, and it does not compare commercial products.
FAQ
What is an equal error rate?
It is the error rate at the threshold where false alarms on real audio and missed fakes happen equally often. It is useful for comparing models, but a real deployment may use a different threshold.
Can a detector trained only on real voices catch new deepfakes?
In this paper it ranked fakes well on familiar data, but its error rate rose to 22.12% on a benchmark with unseen generators and compression.
Does this apply to AI music detectors?
Not directly. The study used speech. The general point, that scores depend on test conditions, is worth asking about for any detector.
Making music with AI tools? See how your track reads on MusicScanner's AI analysis index.
It is a reference signal, not a ruling on how a track was made.