201 of 207: What an AI Music Detector's Blind Test Tells You
An internal blind test where a detector gets 201 of 207 tracks right tells you one thing clearly: on that set, at that threshold, it made six mistakes. It does not tell you what kind of mistakes they were, how the set was split between AI-made and human-made tracks, or how the tool will behave on music it has not seen. MusicScanner's published figure is exactly this kind of number, so this post uses our own result as the example.
What does "201 of 207" actually measure?
It measures accuracy: the share of all classifications that were correct, whether the answer was "AI" or "human". Google's Machine Learning Crash Course defines accuracy this way and notes that every one of these metrics is calculated at a single fixed threshold, so the figure changes when the threshold changes. 201 correct answers out of 207 is 97.1%.
The same guide warns that accuracy is a weak guide when the dataset is imbalanced, or when one kind of mistake costs more than the other. In music review, either condition can apply.
Which mistakes does a single accuracy figure hide?
There are two kinds of error, and they land on different people. A false positive is a human-made track flagged as AI; the false positive rate is the share of all actual negatives classified incorrectly. A false negative is an AI-made track passed as human. For a distributor or a rights society, a false positive can hold up a real artist's release, while a false negative lets an AI track through.
Six errors could be six false positives, six false negatives, or any mix. Accuracy alone cannot tell you which. Some published figures leave this gap: Deezer's announcement, for example, gives 99.8% accuracy and estimates that its detector may miss 0.2% of AI-generated tracks, without a rate for human tracks flagged as AI.
What should you ask before trusting one number?
Five questions turn a single figure into something you can judge. They apply to any vendor's number, ours included.
Question | Why it matters | MusicScanner today |
|---|---|---|
How many items of each class (AI-made vs human-made)? | An imbalanced set can produce high accuracy while one class is handled poorly. | Total published (207). Split not published. |
Which generators and versions, tested when? | Generators change. A test on older versions may not hold for newer ones. | Not published. |
Were the errors false positives or false negatives? | The two errors carry different costs for artists and reviewers. | Six errors. Type not published. |
Were test items kept out of training data, and who held the answer key? | This is what "blind" should mean. Overlap with training data can inflate results. | Described as an internal blind test. Details not published. |
What threshold turned a score into a yes/no? | Accuracy, false positive rate and miss rate all move with the threshold. | Not published. |
What can our 201 of 207 support, then?
It supports a narrow claim. In an internal blind test of 207 tracks, MusicScanner, an AI music detection tool in open beta, classified 201 correctly. It was run by us, not by an independent lab, and the sample is small. That is why we always quote it with those conditions and do not present it as a guarantee about any single track.
It does not support a claim about false positive rates, about any particular generator, or about mixed tracks where a person used AI for part of the work. As the table shows, those are the parts we have not published. If you are evaluating MusicScanner, ask us the same five questions and judge the answers the way you would judge any vendor's.
How should a review team use a detector's result?
Use it to decide which tracks get a closer human look, not to settle a case on its own. When a vendor cannot tell you the split between false positives and false negatives, assume both happen, and design the review step so that a flagged artist has a way to respond.
FAQ
Is 97.1% accuracy (201 of 207 in an internal blind test) good?
It depends on what was tested. The same percentage can come from a balanced set or a lopsided one, and from errors that hurt artists or errors that let AI tracks through. Ask for the class split and the error types before comparing it with another tool's figure.
Is an internal blind test worthless?
No. It shows the tool works on a held-out set under the vendor's conditions. It is a starting point, and an independent or customer-run test on your own catalog tells you more.
Can I compare two vendors' accuracy figures directly?
Only if they were measured on the same kind of data at comparable thresholds. Without that, two accuracy figures describe two different tests.
Evaluating an AI music detection tool? Ask us about MusicScanner.