Skip to content

How do you evaluate an AI voice? Natural is only the beginning

Listen to Story · 5 min listen
Quick Takeaway · Key Summary

An AI voice sounds convincing on first listen. Play the announcement again and you notice that the guest's name is wrong, the time is unclear or the music masks a word. “Does it sound human?” is only one of the questions a broadcast team needs to ask.

A useful evaluation starts with the job. In a short radio link, accurate names and times may matter most. In a long narration, consistency and listening comfort may deserve more attention. Keeping those qualities separate reveals problems that a single overall score can hide.

Give different voices the same script

To compare two voices, start with the same short passage. Include a name, a time, an abbreviation and sentences of different lengths. Keep listening conditions as similar as practical and vary the order in which people hear the samples.

Instead of asking only which voice they prefer, ask what they understood. A listener might write down a name or select it from options; offer appropriate spelling support so a writing difficulty is not mistaken for a hearing problem. Collect opinions about naturalness separately. A pleasant voice can still mispronounce a crucial name.

A small group of colleagues or friends can provide useful preliminary feedback, but it does not represent every listener. Recording who heard which samples and under what conditions makes the result easier to interpret later.

What does a MOS score mean?

In voice comparisons, MOS, or mean opinion score, can refer to an average of listener ratings. The ITU-T P.800.1 recommendation also describes the use of MOS terminology for objective or estimated scores with appropriate identifiers.

📻
RadioTEDU Live Stream

Listen while exploring this story

Tune into campus radio with non-stop independent music and culture.

Imagine two products report MOS values of 4.6 and 4.4. The first looks higher, but if one comes from listeners and the other from a prediction model, treating that gap as proof of a better voice would be misleading. These numbers are illustrative, not product results. In ITU terminology, MOS-LQS denotes a subjective listening score, while MOS-LQO denotes a score predicted by an objective model.

A number by itself does not explain the test behind it. Scores collected with different scripts, devices or listener groups may not be directly comparable. An informal in-house listening session should not be described as standards-compliant unless its methodology actually meets the relevant requirements.

Why a Turkish study uses more than one measure

In a 2026 evaluation of Turkish text-to-speech, Yunus Emre Ozkose and colleagues compare four contemporary systems in different configurations with a baseline system and natural speech. Alongside models that estimate naturalness, they use measures related to intelligibility, signal quality and speaker similarity, and examine consistency over longer speech.

That study cannot by itself choose a voice for a radio station. Its dataset, models and automated measures do not directly establish whether your listeners will understand a guest’s name under your music bed. It does show why a single naturalness score cannot describe the entire listening experience.

Listen with the music in place

A voice that is clear on its own may be difficult to follow once a music bed is added. Listen to the final mix at a comfortable level and pay particular attention to names and numbers. RadioTEDU's RTAI page discusses pronunciation as part of voice preparation and lowering music during speech as part of production.

Revisit difficult examples when the model, voice or script changes. The best voice for a programme is the one that communicates clearly and consistently in that setting, not necessarily the one that wins a general contest for sounding human.

RadioTEDU monthly

The month in podcasts.

Receive podcast episodes from the latest 30 days and upcoming RadioTEDU ticketed events. No daily mail or unrelated announcements.

This consent covers only the monthly podcast newsletter and ticketed event updates. See our privacy notice.

Live RadioTEDU Pick a station and start listening
0:00 0:00

RadioTEDU · Big5 Türkiye Watch Party

RadioTEDU Live Screening: Big5 Türkiye Watch Party. Thursday 19 November, 18.00, D032. Click for free tickets.