Sub-Phonemic Accent Detection: a research case study

Can accent be measured in a phone the recognizer judged correctly labeled? An investigation into acoustic features below the phone-label level — formants, duration, and voice-onset-time — run against 24 non-native and 4 native speakers from L2-ARCTIC and CMU_ARCTIC.

The question

Every measurement the live pipeline reports is a label comparison — did the recognizer output the same ARPAbet symbol as the canonical target sentence calls for? That architecture cannot see accent in a phone judged correctly labeled: a speaker can produce every phone with the right label and still sound audibly non-native, because the label discards how long the phone was held, where its formants sat, and how voicing came in after a stop consonant. This case study asks whether that residual signal is real and measurable, independent of the label-based detector the app already ships.

Method

Phone-level time boundaries don't exist in the live pipeline (it only diffs label sequences), so a plain-numpy Viterbi forced alignment was built over the same wav2vec2 model's own frame-level log-probabilities, targeting each utterance's canonical phone sequence — no new model, no new dependency. The model's CTC output is "peaky" (nearly all probability mass on a blank token, with only 1–2 spike frames per phone), so raw label-to-label frame ranges understate true phone duration 3–5x; boundaries are corrected by redistributing each inter-label blank run at its midpoint, the same fix NeMo/ESPnet's CTC-segmentation recipes use.

From each aligned phone: vowel F1/F2 via LPC (Levinson-Durbin, order 14, steady-state middle 60% of the segment), phone duration, and an autocorrelation-based stop voice-onset-time (VOT) estimate. Vowel formants are Lobanov-normalized per speaker (z-scored against that speaker's own mean/sd) to remove vocal-tract-length differences before any cross-speaker comparison. Each speaker's deviation score is the mean distance from a native reference distribution built leave-one-native-speaker-out — no native speaker is ever compared against a reference that includes their own data.

Correctness is judged by the corpus's own human-annotated ground truth (L2-ARCTIC's annotation_phone_tier_raw field), not by the live pipeline's own judgment — the pipeline's positional precision is ~47% (see the analysis reports), so gating on its own output would inject roughly as much recognizer noise as it excludes mispronunciation noise, confounding the question this study asks.

Headline results

Current numbers (2026-08-23 re-run, 24 L2-ARCTIC speakers / 459 utterances / 14,339 correctly-pronounced phone tokens against a 4-speaker / 125-utterance CMU_ARCTIC native reference):

Voice-onset-time — how long after releasing a /p/, /t/, or /k/ the vocal folds start vibrating — turns out to be the strongest single acoustic signal this pipeline can extract: L2 speakers' VOT deviates from the native reference about 35% more than native speakers deviate from each other, a difference unlikely to be chance (p=0.0044). Phone duration is also significant, though the effect is smaller than an earlier, thinner-sample run suggested (see "how this number moved," below). Vowel timbre — the formant frequencies that distinguish, say, a fronted vs. backed vowel — shows no measurable difference once the sample is large enough to test it properly.

Explore the data

This is a standalone portfolio export — the interactive widgets above run entirely in your browser against data baked into this page; nothing is fetched from any server.

The two widgets below run against the same 919 L2 / 180 native VOT tokens (and the 24 / 4 speaker-level deviation scores) behind the numbers above — nothing here is simulated or pre-rendered. They also make the negative result of a related investigation concrete: a later session asked whether this VOT signal could flag individual mispronounced tokens in the live app, not just a speaker-level group average, and found it couldn't (see VALIDATION.md, "VOT live-flagging feasibility"). Drag the threshold below and watch why.

What doesn't work

Predicting a speaker's first language from these features is at chance — 3 of 24 speakers correctly classified by native-language background (12.5%) against a 16.7% baseline for random guessing among the corpus's 6 L1 groups. Duration, VOT, and formant deviation are real aggregate signals of "accented vs. not," but they don't carry enough information, at this sample size, to say which accent.

How the duration number moved when the sample grew

An earlier run of this same pipeline, on a thinner slice of the corpus (19 speakers, 660 correctly-pronounced tokens), reported duration deviation nearly twice as strong (L2/native ratio ~1.87x, permutation p=0.0011) and flagged vowel-timbre as weak-but-suggestive (p=0.089, just short of significance). Re-running the identical, unmodified pipeline against a 6.5x larger pool (task 036's evaluation-set expansion) cut the duration effect size roughly in half (ratio ~1.27x, still significant at p=0.0316) and flipped the vowel-timbre point estimate below native, erasing its significance entirely (p=0.710). The most likely explanation: 19 speakers at ~35 tokens each is thin enough that a handful of higher-deviation speakers can swing a group mean; at 24 speakers with far more tokens each, that noise shrinks toward the true population value. The lesson generalizes beyond this one study — a sub-phonemic finding built on a thin per-speaker token count should be treated as provisional until re-checked at scale. The vowel-timbre result in particular would have been reported as a real, if weak, effect had this re-check not happened.

Confounds — read before trusting any of the above

• Speaking rate is not controlled. A globally faster or slower talker will show duration deviation on every phone regardless of accent. This is the leading alternative explanation for the duration result and is not ruled out anywhere in this study.
• Recording channel is not controlled. L2-ARCTIC and CMU_ARCTIC were recorded by different labs with different equipment; a systematic channel difference could inflate any native-vs-L2 gap independent of accent.
• The native reference is 4 speakers. CMU_ARCTIC's other voices are themselves non-General-American-accented on purpose, so 4 is a hard ceiling from this corpus, not a sampling choice — it caps statistical power on the native side regardless of how much L2 data is added.
• Speaker sex is only partially controlled (Lobanov normalization corrects vowel formants, but duration and VOT have no equivalent per-speaker rescaling here).

Two follow-up attempts to ship this anyway — both closed out negative

After this case study's group-level result held up, two later sessions each tried a specific, concrete route to making VOT deviation usable as a real per-user signal rather than a research finding. Both are closed out, not left open — here's why.

Averaging over a reading doesn't fix a reference that's too thin to calibrate

One route: score a whole reading's worth of VOT tokens instead of one token at a time, on the theory that averaging out per-token noise (the histogram above shows how much they overlap) would produce a usable speaker-level number even at ordinary reading lengths. Resampling each speaker's own token pool at realistic per-reading sizes confirmed averaging does help, smoothly — from AUC 0.628 at a single token (matching the token-level widget above) up to AUC 0.902 at 20 tokens.

But every one of those numbers is still built against a native reference of the same 4 CMU_ARCTIC speakers this whole case study relies on. Averaging over more tokens shrinks per-token noise; it does nothing about a reference population that's simply too small to trust. A false-positive rate computed this way isn't measuring how a 5th, never-seen native speaker would score — it's measuring how often resampled subsets of the same four people's tokens fall below a threshold also calibrated on those same four people, which is close to circular. The earlier permutation test only needed those 4 speakers to answer "is there a group-level difference at all," a question 4 speakers can support. Gating live output on a specific number is a different, harder question — "is this threshold right" — that 4 speakers cannot answer. What would unblock it: a bigger native General-American reference corpus. A later survey found one concrete, viable candidate (VCTK 0.92, ~21 self-labeled American speakers under a permissive license, automatable with this project's existing tooling) but hasn't pulled or validated it yet.

There is no ground truth to calibrate a VOT threshold against, even in principle

A second route: calibrate a VOT-deviation confidence threshold the same way the live app's label-based detector was calibrated — against independent, human-annotated ground truth. That path doesn't exist for VOT, and not for a data-availability reason more pulling would fix. L2-ARCTIC's human annotators labeled exactly three event types on each phone: substitution, deletion, and epenthesis — all judgments of which phone a listener heard. There is no fourth code, and no separate acoustic measurement, for "this phone's label was correct but its voice-onset-time was deviant." That's not a gap in how this project reads the corpus; it's what the annotators were asked to do, and it excludes by construction the exact case this research track exists to study — a phone whose label is right but whose acoustic realization isn't. The label-based detector could be calibrated against this annotation scheme because "was the label right" is precisely what it recorded. "Was this correctly-labeled phone's timing deviant" is a different, more granular question no annotation in this corpus (or any comparably-sized public accented-speech corpus this project has found) was ever asked to answer. Closing this gap would need a new listening study — human raters judging stop-release timing directly — which is new data collection, not a script against data already on disk.

Reproducing this

src/acoustic_features.py (alignment + feature extraction), scripts/eval_subphonemic_accent.py (corpus pass), scripts/analyze_subphonemic_accent.py (statistics). Full method detail, every intermediate number, and the original thin-sample run's figures (kept for the audit trail) are in VALIDATION.md, "Sub-phonemic acoustic accent detection." The two widgets above fetch subphonemic_widget_data.json, generated from the same two files by scripts/generate_subphonemic_widget_data.py — no number in either widget is computed anywhere but that pipeline.