Accuracy by condition
All eight clips were run through the same model that ships with this page (whisper-base, quantised) on 2026-09-18, and the output was compared word by word against a reference transcript. Every word that came back wrong, missing or added was counted as an error.
- clean synthetic speech — 1 clip, 33 words — 6.1% wrong (2 words)
- real speech, no added noise — 2 clips, 28 words — 10.7% wrong (3 words)
- real speech with light background noise — 2 clips, 57 words — 8.8% wrong (5 words)
- real speech with heavy background noise — 2 clips, 89 words — 33.7% wrong (30 words)
- real speech, telephone band with echo and hiss — 1 clip, 22 words — 22.7% wrong (5 words)
- all eight clips — 229 words — 19.7% wrong (45 words)
The 19.7% is not the average of the five rows. It is 45 error words divided by 229 reference words, so the longer buckets weigh more. Adding the per-bucket percentages and dividing by five gives a different and wrong number.
The worst clip, on its own
One clip accounts for most of the damage. E1-real-noise-heavy is 29.4 seconds of real speech with heavy background noise, 71 reference words, and it came back with 28 of those 71 words wrong — 39.4%. The other heavy-noise clip in the same bucket (E2, 18 words) came back at 11.1%. Same condition, same model, 28 points apart.
That spread is the honest part of this page. A single averaged accuracy figure hides it: if we had only run E2, we would have reported a heavy-noise number four times better than the one E1 actually produced.
What this means if you are about to use the tool
- Clean speech is close to usable as-is. 6.1% on the clean clip, and the two clips with no added noise averaged 10.7%.
- Heavy background noise is where it breaks. 33.7% across the two heavy clips, and 39.4% on the worst one. Expect to read the transcript rather than trust it.
- Telephone-band audio with echo is middling. 22.7% on the one clip in that bucket — worse than clean, better than heavy noise.
- Read the result before you use it. Numbers, names and technical terms are what you should check yourself; those are the words a speech model gets wrong most often.
What we did not test
- Eight clips, not eight hundred. Every percentage above rests on 1–2 clips per bucket. The per-bucket numbers are indicative, not precise.
- One speaker per clip, English. No multi-speaker recordings, no overlapping speech, no accents beyond the ones in these eight clips.
- One noise type. Both noisy buckets use pink noise. Traffic, music and room babble were not tested.
- Chinese was measured separately, on a single 23-second clip, and the model emits Traditional characters where the reference is Simplified — so a raw character comparison is meaningless there. That result is in the dataset, not on this page.
Where the raw data is
Every number on this page comes from two files: wer-by-condition.csv (the five buckets plus the total) and wer-by-sample.csv (all eight clips individually, with the per-clip error counts). Both are published, along with the measurement scripts and the reference transcripts, so the arithmetic above can be checked or re-run.
- Raw data and the measurement scripts — github.com/supersophia8888-cloud/clipquill-asr-benchmark
- Archived copy with a DOI — 10.5281/zenodo.22826968, the concept DOI: that link always resolves to the newest version of this dataset, all versions.
- The same data as a dataset — huggingface.co/datasets/sophia8888/clipquill-asr-benchmark
Run it on your own file
The tool is on the front page of this site. Drop in an audio or video file you already have, pick the spoken language, and read the transcript. Nothing is uploaded — the model runs in your browser tab and the file never leaves your device.