Radio Analyzer: A Severity Scorer That's Honest About Where It Hits a Ceiling
An autonomous pipeline that listens to public police/fire/EMS scanner feeds across 20 US cities, transcribes every call with faster-Whisper, and scores each one 0–10 for notability with a local LLM (qwen3:8b) — hand-judged against real transcripts, including a real finding the eval wasn't designed to flatter: one specific rule in the scoring prompt has a systematic, root-caused, still-unfixed ceiling.
The idea, in one sentence
A police scanner produces more audio than anyone can listen to, so this project transcribes every call in real time and asks a local LLM one narrow question per call — how notable is this, 0 to 10 — so a human only has to read a ranked feed instead of a live audio stream; the interesting part of building it wasn't getting a model to assign plausible-looking numbers, it was finding out exactly where those numbers can be trusted and where they can't.
Does the assigned category and score hold up to an independent read?
24 real calls were sampled (fixed seed, stratified 3-per-category across all 8 scoring categories, 19 different cities) and read independently against the scoring prompt's own stated rubric — not re-run through the model, a genuine second opinion on whether the category matches what the transcript describes and whether the numeric score lands where the rubric's own rules put it. Both axes came back at 87.5% (21/24). The four misses are informative on their own: one infant-respiratory-distress call was scored a routine 5 when the rubric's own "6–8: medical emergency requiring ALS" band applies to exactly that case; a gunshot-wound call naming a victim was scored 6 when the rubric states "a shots-fired call with a victim or address is 8+"; a BOLO/weapon-alert call was mis-categorized as "arrest" though the severity score itself was reasonable either way; and one internally inconsistent case had the model's own summary calling a call "unintelligible" while its category field said "routine." None of the four misses are wild — they're all defensible-looking numbers that a rubric-literal read catches as wrong.
The finding that doesn't get smoothed over: active-pursuit calls are under-scored, and nothing tried so far fixes it
The scoring prompt states its own floor plainly: an active police pursuit is a 9 or 10, regardless of how the rest of the call reads. A follow-up sample of 25 hand-judged chase-category calls that scored below 9 found 10 of 25 (40%) were unambiguous, plain-language active pursuits — "full pursuit of a black male armed with a gun," "we have two vehicles being pursued... eastbound... getting close," a foot pursuit explicitly named as such — scored 6 to 8 instead. The root cause turned out to be bigger than the chase category: across the entire live corpus of 263,743 scored calls, in any category, a score of 9 appears exactly zero times. The model was later confirmed live to be capable of emitting a 9 given a maximally clear synthetic prompt, but in production it caps out at 8 even on textbook active-pursuit language.
Four independently built and tested fixes were rejected, each with real transcript evidence rather than intuition:
| Approach | Result | Why rejected |
|---|---|---|
| Rewrite the scoring prompt's chase rule | Inflated control scores too | A miscategorized assault case moved 6→7 and an already-ended pursuit stayed at 7 instead of dropping — the rewrite couldn't raise real misses without also raising calls that were already correct |
| Narrow second-LLM gate ("is this an active pursuit right now?") | Chase: recall 1.00, false-positive rate 0.70. Shots-fired: recall 0.00 | The gate catches every real miss but also wrongly fires on 7 of 10 calls a human confirmed should stay below 9 — it pattern-matches on pursuit-adjacent words regardless of tense, firing true on "they just lost the vehicle" (the pursuit already over) |
| Pure score-threshold floor (no LLM judgment) | No viable cut point at any threshold | The two groups' score distributions almost completely overlap (see figure above) — the model's own score doesn't carry the pursuit-state signal a threshold would need |
| Cross-call event-sequence context | Doesn't help; would sometimes hurt | All 10 miss cases already state the pursuit unambiguously in the single flagged call; several neighboring calls ("they lost visual," "we got him," "don't chase") would push a context-aware model toward a lower score, not higher |
The common thread across all four: the information needed to tell "genuinely active pursuit" from "pursuit-adjacent language" isn't reliably present in either the model's own score or a differently-framed yes/no question about the same transcript. This is now documented and treated as an accepted model-calibration limitation with the current model, not an open bug being silently re-attempted every session.
A second, messier floor: shots-fired
The same audit applied to the scoring prompt's other explicit floor — "a shots-fired call with a confirmed victim or address is 8+" — found a smaller, less systemic problem: 4 of 25 (16%) clean misses, real confirmed victims or addresses scored under 8. Reading the transcripts shows why it's a different failure mode from chase: the confirming detail is present but stated in genuinely garbled Whisper output of fast radio speech — "515 Avenue Northwest with the Walgreens," "4446 South" — and a narrow LLM gate tested the same way as chase's never fired true on any of these 4 real misses, at any trial. This isn't a scoring-logic gap; it's the same transcript-clarity ceiling under a different name, and the fix (if there is one) more plausibly lives upstream in transcription quality than downstream in the scoring prompt.
What we'd do differently
The honest limits are worth naming plainly. The chase floor has now had four different fix strategies tried and rejected with real evidence, and the most promising untried idea — a pure code-level floor at chase score ≥ 7 → 9 — was scoped but not yet validated against the hand-judged sample; it would trade some new false positives at the 7–8 boundary for a much simpler, more auditable rule than an LLM gate already shown not to discriminate on the dimension that actually matters. The 24-call accuracy sample is a real independent read, but it's still 24 calls; a rule that's wrong 1 time in 20 could easily not appear in a sample that size. And "accepted limitation" is a judgment call, not a proof that no fix exists — it means four specific, reasonable ideas didn't work, not that a fifth one couldn't.
The actual point
An LLM classifier that gets 87.5% on a hand-judged sample is an unremarkable result on its own — the part worth showing is what happened after that number: pulling the thread on a "floor" rule that looked like a small edge case and finding it was a corpus-wide calibration ceiling (score=9 appearing in zero of 263,743 calls), then testing four genuinely different fix strategies against real transcripts instead of stopping at the first plausible-sounding one, and being willing to ship "this is a known limitation" as the actual conclusion rather than a rewritten prompt that looked better on paper but was shown to inflate scores on calls that were already correct.
Reproducibility
Every number on this page is pulled directly from the project's own
metrics.json, eval/accuracy_eval_2026-08-24.json,
and eval/floor_compliance_sample_2026-08-24.json, with the full
investigation narrative (including the rejected-fix evidence) in
eval/README.md and reusable calibration tooling in
eval/calibrate_floor_gate.py — not re-derived here.