Why your word error rate (WER) benchmark might be lying to you

At AssemblyAI, we've spent years helping customers evaluate speech-to-text performance. Word Error Rate (WER) has long been the gold standard metric for this — and for good reason. It's simple, reproducible, and gives you a number you can compare across vendors. But something unexpected happened when we launched Universal-3 Pro, our most advanced transcription model to date. Some customers came back to us saying their benchmarks showed the new model performing worse than our older models. That didn't match anything we were seeing internally. So we dug in. What we found changed how we think about transcription evaluation entirely. How traditional WER benchmarking works Before we get into the issue, let's level-set on the standard evaluation process — because understanding how it works is key to understanding what's breaking down. Here's the typical workflow: Collect 10–20 audio files representative of your core use cases. Submit them to a human transcription provider to get a high-quali