Case study: recalibrating the GOES contrail detector's candidate gate
Synthesizes an eleven-session investigation into one narrative. Every number below is restated from an existing internal artifact and frozen as of when this page was written — this page computes nothing new, and doesn't update itself as later data accrues.
CLOUD_BT_MAX=273.0, BTD_MIN=0.5). Nothing described
below has been applied to production — see “What hasn't happened” at the end.
The question
The shipped GOES detector's only ground-truth-backed accuracy figure — 6.3% precision, 1.9% recall, measured once against a human-labeled satellite validation set and carried forward by assumption to the detectors actually in production — had sat unquestioned for many sessions. Two things made it worth re-examining rather than treating as fixed: recall that low means the detector misses 98 of every 100 real contrails, and nobody had ever asked why.
What was wrong: the candidate gate was too tight
An error analysis (internal: ERROR_ANALYSIS.md) bucketed every false
negative in a 151-scene validation set (40 positive scenes, 19,555 labeled contrail pixels) by which
of the detector's two sequential stages rejected it: a per-pixel spectral candidate gate
(bt13 < CLOUD_BT_MAX and btd > BTD_MIN, deliberately loose — meant to cut
obvious non-cloud pixels cheaply before the expensive step) followed by an anisotropy (line-shape)
threshold. The finding: most labeled contrail pixels were being rejected at the
first, supposedly loose stage, before the line-detection logic ever got to evaluate
them. The gate meant to be a cheap pre-filter was doing most of the false-negative damage. That
pointed at the candidate-gate thresholds — not the line-detection threshold — as the
parameters to retune.
What was proposed
A grid search over the candidate gate (internal: CANDIDATE_GATE_SWEEP.md) found a joint optimum on the same 151-scene population the error analysis used:
| Configuration parameter or metric | Shipped | Proposed |
|---|---|---|
CLOUD_BT_MAX | 273.0 K | 300.0 K |
BTD_MIN | 0.5 K | 2.0 K |
LINE_Z_MIN | 1.0 σ (unchanged) | 1.0 σ |
| Precision | 6.30% | 10.67% |
| Recall | 1.94% | 13.73% |
| F1 | 0.0296 | 0.1201 |
The recall gain's 95% bootstrap confidence intervals don't overlap (shipped [0.74%, 3.24%] vs. proposed [9.55%, 18.57%]). The precision gain's intervals overlap slightly (shipped [2.03%, 11.29%] vs. proposed [6.13%, 15.67%]) — directionally positive but not independently significant on 40 positive scenes.
A proposal that only beats shipped on the data it was tuned on is a common failure mode (overfitting the tuning sample), so the recommendation was treated as unproven until checked three more ways.
Check 1 — does it hold on data the search never saw?
A held-out validation pass (internal: HELDOUT_VALIDATION.md) re-scored both configurations on a disjoint 151-scene batch the grid search had never touched.
| Metric | Shipped in-sample | Shipped held-out | Proposed in-sample | Proposed held-out |
|---|---|---|---|---|
| Precision | 6.30% | 2.80% | 10.67% | 7.70% |
| Recall | 1.94% | 1.07% | 13.73% | 11.50% |
| F1 | 0.0296 | 0.0154 | 0.1201 | 0.0923 |
Both configurations score lower in absolute terms on unseen data (expected — some in-sample-fit inflation is the normal cost of tuning and evaluating on the same data). What matters is the ordering: proposed still beats shipped on precision, recall, and F1 on data it was never fit to. The recall gain's confidence intervals still don't overlap held-out (shipped [0.37%, 1.87%] vs. proposed [8.26%, 14.55%]) — the statistically robust part of the claim survived the held-out test intact, now validated across 302 scenes total.
Check 2 — is the low precision explained by cirrus contamination instead?
A separate, older open question asked whether the shipped detector's low 6.3% precision was mostly false alarms on natural cirrus (which produces the same spectral signature as a contrail but has no aircraft behind it). A negative-control test (internal: NEGATIVE_CONTROL.md) checked this with real ADS-B flight tracks: if cirrus contamination dominated, positive detections should lack a plausible nearby aircraft more often than clear pixels. Across a single day, a 4-day pooled expansion, an 8-city replication on a disjoint roster, and an orientation-aware tightening of the match criterion, the result was consistently null or reversed: positive detections were never more likely to lack a plausible aircraft than clear pixels (e.g. pooled: 6.0% vs. 5.1%, Fisher p=0.80; West replication: 0.0% vs. 7.9%, p=0.16).
This doesn't explain why the proposed configuration is better — it rules out a specific alternative explanation for the shipped configuration's low precision, so the retune recommendation isn't competing against an unaddressed rival theory.
Check 3 — does it hold up on live, ongoing data?
Both prior checks were frozen, one-off snapshots of the same static archive. A shadow-mode calibration check (internal: SHADOW_CALIBRATION.md) built a live third check: a shadow scorer runs daily alongside the shipped scorer, scoring both the shipped and proposed configurations against that day's real, newly-fetched satellite scenes at the same city-hours (paired by city and time). A trend report reconstructs the gap per day rather than only cumulatively, so a single lucky pooled sample can't hide a trend that isn't real.
As of the trend report frozen for this write-up (6 accrued days, Aug 19–24, 2026; 132 paired city-hours/day, 792 total):
| Date | Shipped detect rate | Proposed detect rate | Gap (pp) |
|---|---|---|---|
| Day 1 | 38.6% | 87.9% | +49.2 |
| Day 2 | 46.2% | 88.6% | +42.4 |
| Day 3 | 39.4% | 90.9% | +51.5 |
| Day 4 | 53.8% | 92.4% | +38.6 |
| Day 5 | 46.2% | 86.4% | +40.2 |
| Day 6 | 42.4% | 90.2% | +47.7 |
| Cumulative | 44.4% | 89.4% | +45.0 |
The gap has pointed the same direction on every single accrued day (6/6) — directly corroborating the static backtests' recall-gain finding on real, ongoing data rather than a frozen archive.
This is not the same claim as “precision/recall improved.” Live satellite scenes have no per-pixel human label, so the shadow scorecard cannot compute precision/recall the way the archive-based backtests do. It measures a different, live quantity — does the proposed configuration keep detecting more than the shipped one on real ongoing city-hours — and answers yes, consistently, so far.
What this adds up to
Four independent checks, four different mechanisms, all pointing the same direction:
- Diagnosis — the candidate gate, not the line-detection threshold, is what's suppressing recall.
- In-sample optimum — retuning it roughly doubles precision and recall (6.3%→10.7%, 1.9%→13.7%).
- Held-out replication — the ordering survives on data the search never saw (302 scenes total).
- Live replication — the same ordering shows up on 6/6 accrued days of real, ongoing satellite data (792 paired city-hours), not just frozen archives.
Alongside this, a plausible alternative explanation for the shipped configuration's low precision (cirrus contamination) was tested directly and did not hold up — so the retune isn't competing against an unaddressed rival theory for what's actually wrong.
Honest caveats — read before acting on this
- The recall gain is the strong part of the claim; the precision gain is not. On both the in-sample and held-out backtests, the precision confidence intervals overlap (shipped and proposed are not statistically distinguishable on precision alone at 40–61 positive scenes) — only the recall gain's intervals are non-overlapping on both populations. Don't read “roughly doubles precision” as equally certain to “roughly doubles recall.”
- Six days of shadow-mode data was a short trend at the time this snapshot was frozen. Six consecutive days pointing the same direction is corroborating, not conclusive — the same caution this project applies everywhere else to small accruing samples. The shadow scorecard keeps accruing past this snapshot; a longer run is a natural next check, not something this static page updates on its own.
- The shadow scorecard measures a different thing than the static backtests. It cannot compute true precision/recall (no per-pixel ground truth on live satellite scenes) — it only shows that the proposed configuration keeps firing more often than shipped on real data, which is consistent with (but not independent proof of) the recall gain the backtests measured directly.
- The negative control's null result is itself not a strong exoneration. Single-day and multi-day samples are small, ADS-B ground coverage is incomplete, and the match methodology has a published attribution ceiling even for the best-resourced teams. It rules out cirrus contamination as the dominant driver; it doesn't prove it contributes zero.
What hasn't happened
Applying this recommendation is a deliberate, multi-step, owner-triggered change (re-point the two constants in the detector, regenerate the calibration artifact, regenerate the provenance claims list, regenerate the public forecast and pro-draft pages). Nothing in this project's public output — the live forecast, the pro concept draft, or any claim this project's provenance checker verifies — reflects the proposed numbers. The shipped detector still runs its original thresholds. Flipping the headline to the proposed numbers remains the owner's call, not something decided here. This page exists so that decision, whenever it's made, doesn't require re-reading eleven sessions of prior work first.