Accuracy Report

What this page is

Every number below was produced by Lucy’s own evaluation tool (lucy-eval) running the same transcription and speaker-attribution code the app ships, on public corpora anyone can download. Nothing here is a vendor benchmark, and nothing here appears in the App Store listing — a listing claim has to hold for every phone and every room, and these numbers hold for the conditions stated next to them.

WER (word error rate) is the share of words that were substituted, dropped or invented, over the reference transcript. CER is the same at character level, used where words are not the unit of writing. DER (diarization error rate) is the share of speech time assigned to the wrong speaker, missed, or invented. Lower is better for all three.

Transcription — word error rate by language

Median WER over each language corpus, three Whisper model sizes. Lucy names them Draft (base), Balanced (small) and Highest quality (large-v3-turbo). Which one runs is decided by the device: iPhones and Macs with 8 GB or more run Highest quality; 6 GB iPhones run Balanced; A13-class phones (iPhone 11, SE 2nd generation) run Balanced or Draft. Measured 2026-08-03 on the production pipeline.

† The Spanish corpus has 3 clips, not 15; its 0.0% is a small-sample artifact, not evidence that Spanish is solved. Swiss German is unmeasured.

What the table says. Highest quality is best or tied in all 15 measurable languages and never loses; the margins are often large (Dutch 23.8% to 4.8%, Turkish 27.3% to 7.7%). Balanced is not a safe default: in Korean it is worse than Draft, and in Russian it ties. Draft is unusable for several languages — Hindi above 100% means the model emitted more wrong words than the reference contains, and Romanian, Dutch and Ukrainian sit between 45% and 55%. This is why Lucy picks the largest model the device can hold, and says so on the download screen instead of hiding the choice.

Japanese and Chinese — character error rate

Word-level WER is structurally invalid for these languages (there are no spaces to split on, so one wrong character fails a whole clause), and it reported 100% for models that were in fact working. The tool now switches to character error rate for them automatically.

‡ The Chinese Balanced cell is missing because the model failed to load during that run; the cause was not established and the cell was left empty rather than guessed.

On the phone versus on a Mac

The same English clip (833 reference words) transcribed on an iPhone 14 Pro Max scored a WER of 3.7% against 3.5% on a Mac with the same model — the phone runs the same models with the same accuracy, only more slowly. One measured exception is worth stating: on an A13-class iPhone, which runs the smaller models, a single French device run measured markedly worse than the Balanced column above, and that result is still under investigation. If you record in a language other than English on an iPhone 11 or SE, read the Balanced column as the best case.

Speaker attribution — diarization error rate

Measured with the streaming speaker-attribution backend Lucy uses for recordings, on two public meeting corpora: VoxConverse (two-person conversations) and the AMI Meeting Corpus recorded through a single distant microphone (four-person business meetings, the hardest common case). Measured 2026-05-11 and reproduced by the current tool.

What the table says. Two people at a table are labelled almost perfectly. Four people talking over each other through one distant microphone are not: in the harder AMI sessions, roughly one word in five is attributed to the wrong person. That is the honest shape of single-microphone recording, and it is why the app says “up to 4 voices” rather than promising more. On a real six-person meeting recorded on a phone, the uncapped backend Lucy uses for imported files recovered five of the six voices; the sixth was acoustically merged at every setting we tried. Expect four, sometimes five, from a phone on the table.

Summaries

Summaries are generated on the device by a 4-billion-parameter model. Every summary line must cite a span of the transcript; lines that cite nothing are dropped before you see them, and lines whose numbers, names or verbs do not match their cited span are shown with a warning. Nothing is saved until you accept it. On a real 26-minute, six-person meeting, about one line in thirteen carried that warning — flagged means “could not be verified word for word”, not “wrong”, and it is exactly the line you should read against the recording.

Method

• Tool: lucy-eval, the evaluation command built from the same Swift package as the app, with no acoustic pre-processing beyond what the app does.
• Transcription models: WhisperKit openai_whisper-base, openai_whisper-small, openai_whisper-large-v3-v20240930_turbo.
• Transcription corpora: FLEURS test split (CC-BY-4.0) for most languages; VoxPopuli and LibriSpeech for German, English, Spanish, French and Romanian. 15 clips per language unless stated. Reported figure: median WER over the corpus.
• Speaker attribution: FluidAudio Sortformer v2.1 (streaming) for the table above; imported files use the offline pyannote community-1 backend, which has no four-speaker ceiling.
• Corpus integrity gate: an earlier run scored eleven corpora whose audio had decoded to silence; the model hallucinated over the silence and reported a confident, plausible, completely wrong number. Every corpus is now proven non-silent, paired with reference text and at 16 kHz before any evaluation starts, with a peak-amplitude threshold rather than a zero test, because the decode fault produced peaks of 0.000031.
• Not measured here: battery use, processing time on older phones, accented non-native speech, phone-quality audio, five or more speakers. Device-class measurements will be added as they are made.
• Reproduce: bash Corpus/scripts/run_wer_matrix.sh in the evaluation harness; results are committed next to the run that produced them.

Questions, or a corpus you think should be added? support@xymex.com