Parts-Risk Prediction at Scale: From 10% to 60% Recall

A legacy heuristic was catching roughly 1 in 10 aircraft parts actually at risk of stockout across a fleet of 1M+ part numbers. Replacing it meant treating stockout risk as what it actually is — a time-to-event problem, not a single yes/no label.

10% → 60%recall of at-risk parts vs. the prior heuristic
6×improvement in recall
1M+aircraft part numbers scored across multiple fleets

The problem, in one sentence

A legacy heuristic was catching about 1 in 10 aircraft parts actually at risk of stockout across a fleet of 1M+ part numbers — the other 9 were found the hard way, on the flight line. The task was to build a system that could be trusted to catch most of them ahead of time.

Why this is harder than a single classifier

Stockout risk isn't really a single event to classify — a part doesn't just "fail," it fails by some date, and usage histories across a fleet this size are routinely incomplete or still ongoing (censored) at the time a prediction is needed. Framing the problem purely as classification throws away that timing signal; framing it purely as survival analysis throws away the discrete, categorical failure-mode signal a classifier captures well. The system built here combines both: a survival-analysis model (xgboost-survival) for time-to-stockout, ensembled with a classification model capturing discrete risk factors, scored across 1M+ part numbers spanning multiple fleets.

The result

Recall of at-risk parts improved from roughly 10% to roughly 60% against the prior heuristic it replaced — a 6× improvement, at fleet-wide, 1M+ part-number scale.

A retrieval layer on top: answering "why is this part at risk"

A risk score alone doesn't tell a technician why a part is flagged, so a second system sits on top of the model: retrieval-augmented question answering over unstructured maintenance records, letting a technician ask a plain-language question instead of keyword-searching decades of scanned maintenance manuals. Two parallel retrieval implementations were built — one on LlamaIndex + pgvector, one on AWS Bedrock. The manuals themselves were digitized with an OCR pipeline (Tesseract + AWS Textract) to make decades of scanned paper searchable at all.

Constraints and what isn't shown here

The system was deployed on Advana (DoD's enterprise data platform) for the U.S. Air Force, under strict security and data-governance constraints. The account above stays at the level of the published architecture and the headline metric; the specific engineering trade-offs behind that architecture — including exactly why two separate retrieval stacks exist rather than one, and the concrete shape of the governance constraints themselves — aren't detailed on this page, because they aren't public. That's a real limitation of this write-up, named plainly rather than papered over with generic language about "strict security requirements."

Reproducibility

Unlike this site's other case studies, there is no public repository, writeup, or reproduction script for this project — the work was built under a government contract with no publicly shareable codebase. The figures on this page (the recall improvement and the part-number scale) are the same ones published on this project's card on the portfolio homepage.