XGBSE Revival: Survival Analysis Benchmark

Revived an archived survival-analysis library, extended it with five new estimators, and then ran the kind of adversarial benchmark designed to embarrass it rather than flatter it — reporting the honest, unflattering headline alongside the real, narrower edge that survived scrutiny.

9survival estimators (4 revived + 5 new)

13models benchmarked

15datasets (12 synthetic/canonical + 3 real clinical)

117/117package tests passing

7th of 13best xgbse rank, 12-dataset aggregate

The headline

xgboost-survival-embeddings (xgbse) is a survival-analysis package built on top of XGBoost, archived by its maintainers on 2024-10-03 with dependency pins already behind the major-version bumps both XGBoost and numpy had shipped by the time this project found it. After forking it onto the current numpy/xgboost/scikit-learn/pandas stack, reading the source closely enough to build five new estimators on its internal two-stage pattern (a boosted tree feeding a classical curve-fitting stage), the most adversarial benchmark this project could design was run against it: Cox models, Random Survival Forests, gradient boosting, and raw XGBoost survival regression, across synthetic data, canonical biostatistics datasets, and three real clinical cohorts — with tuning passes and paired significance tests layered on specifically to try to overturn the result, not confirm it.

The honest headline: on aggregate, xgbse's own curve-generating machinery does not beat plain XGBoost or a well-chosen classical baseline on discrimination. But that headline inverts on two of three real clinical datasets, survives honest hyperparameter tuning in both directions, and the package's genuine edge — a calibrated survival curve at a specific speed/scale niche — turned out to be real, just narrower and more conditional than a promotional summary would claim.

Stage 1 mean rank across 12 datasets — classical baselines and raw XGBoost AFT sweep the top six spots; the best xgbse variant ranks 7th of 13 — On the 12-dataset synthetic/canonical aggregate, classical baselines and raw XGBoost AFT sweep the top six spots. The best xgbse curve variant, `DebiasedBCE`, ranks 7th of 13 by mean per-dataset rank.

Speed: true against one comparison, false against another

xgbse's usual pitch is speed. That claim needed a number, not a slogan, so the geometric-mean ratio of fit CPU-time (core-seconds, not wall-clock — a shared host machine's OpenMP thread contention produced an impossible first-pass reading where fit time fell as data grew, until re-measured with RUSAGE_SELF+CHILDREN and a contamination detector) was computed between xgbse's variants and each baseline family, across a synthetic sweep from 1,000 to 32,000 rows:

vs. baseline family	ratio (xgbse / family)	reading
Random Survival Forest	0.13×	~7.7× cheaper
Gradient Boosting (sksurv)	0.28×	~3.6× cheaper
raw XGBoost AFT (native)	6.92×	~7× costlier
Cox / Coxnet	11.57×	~12× costlier (Cox is trivially cheap regardless)

So "xgbse is cheap to fit" is true against the tree-ensemble baselines it's usually pitched against, and false against the plain booster it wraps and against Cox. One variant, KaplanNeighbors, is the exception worth naming: at n≥8,000 rows it's the cheapest fit of all 13 models compared, cheaper even than native XGBoost AFT.

Speed vs accuracy at n=8000 — every xgbse survival-curve variant is Pareto-dominated on accuracy; its real edge is fit cost, not peak discrimination — Every xgbse survival-curve variant is Pareto-dominated on accuracy at n=8,000 — its real edge is fit cost, not peak C-index.

The twist: real clinical data inverts the ranking, and it isn't uniform

Every dataset where xgbse's family had beaten Cox up to this point was synthetic or small (veterans, n=137) — the weakest kind of evidence. Run on three real clinical cohorts — SUPPORT (n=8,873), METABRIC (n=1,904), and WHAS (n=1,638) — the aggregate ranking inverts on two of the three. On WHAS specifically, xgbse.DebiasedBCE wins outright: #1 of 14 on discrimination (C=0.8690) and the best calibration in the field (IBS=0.0807). On METABRIC, Cox does not lose (2nd of 14) — but that dataset's entire model field spans only 1.5× the average per-model fold noise (vs. 7.4× on SUPPORT and 5.0× on WHAS), so the honest reading is that METABRIC barely separates any model from any other, not that Cox specifically wins it.

Real-data significance forest plot — paired 95% confidence intervals on fold-level differences across WHAS, METABRIC, and SUPPORT — Paired 95% confidence intervals on fold-level differences, all three real datasets sharing identical CV folds. RSF and `DebiasedXGB` beat CoxPH with CIs excluding zero on both WHAS and SUPPORT; every headline comparison on METABRIC straddles zero.

It survives tuning, in both directions

A validation-selected tuning pass cuts against Cox either way: on the synthetic aggregate, tuning gives xgbse's own variants the largest gains but closes only about a third of the gap to the baselines. On the real clinical sets, Cox turns out to be near-untunable (Δ ∈ [−0.0005, +0.0008]), while the tree- and embedding-based models gain 0.01–0.03 C-index and the Cox-trails gap widens rather than narrows.

The most instructive failure

The third new variant, DebiasedQuantile, finished dead last in every stage — a 0.635 mean C-index against the field's 0.75–0.80. Diagnosing why turned out to be the most useful part of the project: it fits per-time-bin quantile regressors against a binary event indicator, but the α-quantile of a Bernoulli(p) variable only ever takes one of two values — meaning the per-bin hazard estimate has exactly 2 attainable values, versus 228 for DebiasedBCE's logistic regression on the same bins. That's a design flaw, not a tuning gap, and it's fixable by acting on the diagnosis: a redesigned variant regressing IPCW-weighted continuous event-time quantiles instead of the binary target restores resolution (2 → 71+ attainable values per bin) and recovers a mean +0.064 C-index across the three real datasets.

Two bugs a passing test suite didn't catch

Getting the dev-checkout test suite green after modernizing looked cheap — one removed test helper, a handful of pandas deprecation fixes, zero code changes needed for numpy 2.x or xgboost 3.x. That verdict didn't hold once the actual distributable wheel was built and tested in an isolated environment, independent of the dev checkout:

xgboost≥3.4.0 broke one estimator's fit(), and staying pinned wasn't the end of it. A sorted-array assumption change in xgboost's quantile-regression objective caused a 100%-reproducible crash; fixing the crash surfaced a second, deeper regression (concordance dropping to chance, ~0.496) traced to xgboost's auto-initialized base_score landing on a domain boundary for this estimator's binary target. Setting base_score=0.5 explicitly restored correct behavior (0.4963 → 0.5478 on the exact test fixture), verified with no regression under xgboost<3.4.0.
A caret-pin trap silently broke numpy 2 support. The original lifelines = "^0.29.0" pin capped at <0.30.0 under Poetry's pre-1.0 semantics — and PyPI has nothing between the numpy-<2-only lifelines<=0.29.0 and the numpy-2-supporting lifelines>=0.30.0. The built wheel was uninstallable in any numpy-2 environment despite every dev-suite test passing.

Neither bug showed up in the editable-install test suite, because that venv already had a working dependency combination installed before either package was ever built from scratch — exactly the kind of gap that only building and installing the real artifact exposes.

When to actually reach for xgbse

You need a calibrated survival curve, not just a risk ranking — nothing in this benchmark beats xgbse's curve variants on that specific job, because nothing else compared does it at all.
Large-n data (≥~8,000 rows) where fit cost matters — the simple variants sit on the compute-cost Pareto frontier; the sophisticated per-bin variants do not.
You want prediction intervals on survival time — the redesigned DebiasedQuantile is the only variant in the comparison that produces one.
Otherwise, for peak discrimination on small or roughly proportional-hazards data: use Cox or raw XGBoost AFT. No xgbse curve variant beats them in this benchmark.

Reproducibility

Every number on this page traces to a partial:false benchmark JSON committed alongside the project's code, and every figure was regenerated directly from those JSONs by a source script, not a static export. A separate adversarial re-analysis script, deliberately written without importing any of the project's own reporting code, re-derived the load-bearing claims from scratch and found zero numeric discrepancies. The package's test suite (117/117) passes on Python 3.9 through 3.13, and the actual built wheel — not just the editable dev checkout — installs and smoke-tests clean in an isolated environment.

Case study for the XGBSE Revival project. Content condensed from the project's own ARTICLE.md — every number above is quoted or paraphrased directly from that source, not re-derived here.