Detecting data drifts in dynamical systems

A structural-change detector that works zero-shot on series it has never seen: 42 statistical features distilled from a 170+ candidate pool feed a 3-2-1 stack of heterogeneous learners (TabPFN, XGBoost, Random Forest, CatBoost, LightGBM), fused interpretably with RuleFit and wrapped in a POMDP layer that turns pointwise scores into a regime belief.

Every changepoint detector, from CUSUM and Page-Hinkley to a classifier trained on labeled breaks, quietly learns the idiosyncrasies of the series it was tuned on: "volatility above X means trouble" on data where that happened to be true. Then the regime shifts, the shortcut stops mapping to reality, and the detector is most confident exactly when it should be least. A detector meant to survive regime change can't learn any single domain's quirks; it has to learn what a break looks like, an invariant signature, and nothing else.

I'd been calibrating discretionary tools against single-domain models, an XGBoost here, a random forest there, and it worked until the generating process moved, precisely the moment I needed a trustworthy answer. Retraining per regime is circular: detecting the regime change is what tells you to retrain.

The push came from the ADIA Lab Structural Break Challenge, specifically the write-up by the winning Alphabot team, who took first with a stack of diverse first-level tree models merged by a meta-model. Their result validated the core intuition: no single tree architecture is unbiased, but their biases differ, and stacking distinct statistical perspectives beats tuning any one. I wanted to add three things on top: a deliberately heterogeneous training corpus (financial data, synthetic assets from delta replication, 12-lead ECG, weather, energy) so the model applies zero-shot without retraining; interpretable fusion via RuleFit instead of an opaque meta-model; and a sequential decision layer, a POMDP, turning pointwise scores into a belief about the current regime. The output is a calibrated structural-change score used upstream of decision tools as a confidence throttle, but everything below is about the machine, not the use case.

01How it works

Four stages: a feature distillery, a 3-2-1 stacked ensemble, a RuleFit fusion layer, and a POMDP temporal layer. Flow: raw series -> causal rolling windows -> 42-dimensional feature vector -> layer-1 learners (TabPFN, XGBoost, Random Forest) -> layer-2 meta-learners (CatBoost, LightGBM) -> layer-3 reducer (single shallow GBT) -> rule extraction -> sparse linear rule model -> per-window score -> POMDP belief update -> regime belief and throttle action.

Feature distillery: 170+ -> 42

Each window is summarized by descriptors from a candidate pool of 170+, in nine families:

1. Distributional: moments, quantile spreads, tail indices, mode multiplicity.
2. Autocorrelation / linear structure: ACF/PACF summaries, AR(p) fit coefficients and residual diagnostics.
3. Spectral: band energies, spectral centroid and flatness, spectral entropy.
4. Complexity / entropy: permutation entropy, sample entropy, Lempel-Ziv complexity.
5. Recurrence: recurrence quantification: rate, determinism, laminarity, trapping time.
6. Scaling: Hurst-type exponents via DFA and R/S, multifractal width.
7. Stationarity statistics: rolling-split test statistics between window halves.
8. Wavelet: energy ratios across dyadic scales, wavelet leaders.
9. Model-comparison: likelihood gaps between simple competing generative fits, in the spirit of catch22's "which model explains this best."

The pool is filtered along six axes, each a hard gate with an explicit test and pass criterion:

Selection then runs three passes: the hard gates cut the raw pool; stability selection recomputes the probe-ensemble ranking across bootstrap resamples of training domains, keeping only features selected in most runs; a greedy mRMR-style forward pass grows the final set until marginal gain on held-out series drops below epsilon. Forty-two features survive.

Before the ensemble, the 42 are residualized against slow trend, rank-normalized causally (past windows only), and whitened, letting a barometric pressure series and an order-flow series share one feature space. Level and scale are deliberately destroyed; only shape and dynamics remain.

The 3-2-1 stack

Layer 1 holds three learners chosen for maximally decorrelated inductive biases:

• TabPFN: a prior-fitted transformer doing Bayesian-flavored in-context learning; strong on small, clean windows, with error modes unlike any tree. The 42-feature budget keeps every window inside its input constraints.
• XGBoost: sequential bias reduction, sharp axis-aligned splits, level-wise growth.
• Random Forest: variance reduction by bagging, smooth and hard to destabilize.

Each layer-1 learner emits out-of-fold probabilities under a purged, embargoed blocked scheme (per Advances in Financial Machine Learning). Purging is mandatory because overlapping rolling windows make naive K-fold leak: a window ending at t shares raw samples with one starting at t-k in another fold. Any training window overlapping a test block plus embargo margin is dropped; layer 2 only ever sees out-of-fold probabilities.

Layer 2 is two meta-learners trained on layer-1 probabilities plus a compressed slice of raw features, learning when to trust whom:

• CatBoost: ordered boosting and ordered target statistics make it the least leakage-prone booster available, which matters more at the meta level than anywhere else.
• LightGBM: leaf-wise growth with histogram binning, carving deep narrow interaction paths level-wise XGBoost tends to miss, keeping layer 2's members decorrelated.

Layer 3 is a single shallow GBT reducing the stack to one per-window score. The narrowing shape (3 -> 2 -> 1) is the point: width at the bottom buys disagreement, each layer arbitrates the previous one's disagreements, shallow top capacity forces simple arbitration. The layer-1 disagreement itself, variance across the three probability streams, turned out to be one of the most informative meta-features: heterogeneous models diverge hardest near a structural break, before any is individually confident.

RuleFit fusion

Instead of an opaque scorer, every tree is decomposed into root-to-leaf paths, each a binary rule ("permutation-entropy delta > a AND determinism drop > b"). Following Friedman & Popescu's RuleFit, an L1-penalized linear model fits over the rule indicators plus winsorized linear terms of the 42 features. The lasso keeps a few dozen rules out of thousands; the fused detector is a weighted rule list.

This buys interpretability with teeth (partial-dependence checks run against the rule list directly, and a rule whose support is dominated by one domain flags shortcut learning; several were killed this way) and regularization (L1 fusion prunes split structure that only ever fired on one domain's quirks).

The POMDP temporal layer

A per-window score is a pointwise judgment; regime detection is sequential, and the regime itself is never directly observed, only its statistical shadow. That's a POMDP by definition, and the quickest-change-detection literature has long framed Bayesian break detection exactly this way.

The latent state space is a small set of regime states (stable, transitional, broken, plus break-type refinements); the observation is the fused RuleFit score with emission densities per state estimated on training domains; the transition prior is sticky, encoding regime persistence. A Bayes filter updates belief after every window. Actions are discrete confidence-throttle levels, reward trades detection delay against false-alarm cost, so the operating point is chosen by policy, not an ad hoc threshold; the policy is solved offline on the discretized belief simplex. Payoff: isolated score spikes get absorbed instead of triggering, and sustained moderate scores accumulate into detection even when no single window is decisive.

Training for invariance: the corpus is the regularizer

The corpus spans five dissimilar domains: financial market data, synthetic assets from delta replication (market-like, generating process known and controllable), 12-lead ECG, weather, and energy series. These domains share nothing at the level of units, scale, sampling regime, or surface statistics, so any split rule keying on a market idiosyncrasy is penalized by the ECG/weather portions of the loss, and vice versa. The only structure that pays off everywhere is what all five domains genuinely have in common: what a shift in underlying dynamics looks like once level and scale are stripped away. The corpus doesn't tell the model what a structural break is; it removes every cheaper thing the model could learn instead.

Labels combine synthetic changepoints (injected mean/variance/spectral shifts and parameter drifts, the same break taxonomy the ADIA Lab data covered) with annotated real-world breaks in the style of the Turing Change Point Dataset; the delta-replicative synthetic assets are especially useful since breaks can be injected into a market-like process with exact ground truth. Validation is always on held-out series never trained on, drawn from every domain, with cross-domain consistency, not any single domain's score, as the selection criterion. Target classes: level/variance breaks, phase shifts, observability loss, inflation of aleatoric uncertainty.

02Trade-offs

03Validation, and why there are no numbers here

I'm deliberately not publishing benchmark tables or comparative statistics: the detector runs upstream of live trading tools, and its measured operating characteristics are part of that edge. What I can share is the methodology.

All evaluation is on held-out series never trained on, drawn from every domain, with the acid test being the non-financial ones: a detector that scores well on held-out market data proves little (the overfit trap from the intro); one that scores consistently on held-out ECG, weather, and energy series, using the exact same frozen model, evidences invariance. Performance is tracked as break/no-break discrimination on labeled windows plus detection-delay vs. false-alarm trade-off against classical online detectors (BOCPD, Page-Hinkley, ADWIN) and against each layer-1 learner solo. Two findings shaped the design: the stack's advantage over its best single member is largest on the domains furthest from finance, and the metric governing every decision was worst-domain performance, not average. A domain-agnostic claim lives or dies on its minimum, not its mean.

04Where it went sideways

The psychic detector. The first end-to-end run produced held-out AUROC numbers too good to be real. The culprit: rank-normalizing each feature over the full series meant every pre-break window's percentile was computed against statistics that included the post-break distribution. The detector wasn't detecting breaks, it was reading them out of the normalizer, invisible in a backtest, since the numbers just look great. The fix: strictly causal rolling normalization (expanding or trailing window, past data only), plus a standing invariant test: recompute every feature on the series truncated at time t and assert bit-equality with the streaming value at t.

Redundancy that wasn't. Early feature screening clustered redundant features on the pooled cross-domain correlation matrix and kept one per cluster. Two spectral features ~0.97 correlated on market data (and thus collapsed to one) turned out nearly independent on quasi-periodic physiological signals, and the discarded one carried most of the phase-shift signal there. Held-out performance on physiological data cratered while every other domain looked fine, exactly the signature cross-domain validation exists to catch. Fix: per-domain-then-intersect redundancy screening, a pair is "redundant" only if redundant everywhere.

05What I'd do better next time

The ensemble architecture wasn't where the difficulty lived; the 3-2-1 stack, RuleFit fusion, and POMDP layer went in mostly as designed. The real constraint, the thing every failure traced back to, was the feature distillery and its evaluation protocol: causality of every transform, per-domain validity of every screening decision, leak-freedom of every meta-feature. The model learns whatever the pipeline lets it, and every shortcut it found was one the pipeline had left open.

Two directions for next version: treat layer-1 disagreement as a first-class signal, modeling the distribution of cross-learner divergence explicitly since it front-runs individual detectors near breaks, feeding it to the POMDP as a second observation channel. And close the loop between RuleFit and feature screening: rule-support-per-domain is the sharpest shortcut detector in the system and currently runs as manual audit; automating it, rejecting features whose derived rules concentrate on one domain during selection rather than after fusion, would turn the interpretability layer into an active regularizer. Interpretability wasn't the deliverable, it was the debugger.

06Further reading

• ADIA Lab Structural Break Challenge: the competition that framed the problem: tens of thousands of labeled univariate series with heterogeneous break types.
• How far can we push the winning model of the ADIA Lab Structural Break Challenge? (Brandao et al.): the winning Alphabot team's write-up; the diverse-stack-plus-meta-model design that inspired this project.
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second: the prior-fitted network in layer 1; its input constraints shaped the 42-feature budget.
• CatBoost: unbiased boosting with categorical features: ordered boosting, the leakage-resistance argument for using it at the meta level.
• LightGBM: A Highly Efficient Gradient Boosting Decision Tree: leaf-wise growth, layer 2's second member.
• XGBoost: A Scalable Tree Boosting System: layer-1 boosting learner.
• Predictive Learning via Rule Ensembles (Friedman & Popescu): the RuleFit fusion.
• Quickest Change Detection (Veeravalli & Banerjee): the sequential-detection framing behind the POMDP layer.
• catch22: CAnonical Time-series CHaracteristics: the feature-distillation philosophy this pipeline extends toward break sensitivity.
• Permutation Entropy: A Natural Complexity Measure for Time Series (Bandt & Pompe): core dynamics feature.
• Recurrence Plots for the Analysis of Complex Systems (Marwan et al.): recurrence quantification features.
• An Evaluation of Change Point Detection Algorithms (van den Burg & Williams): real-world changepoint labels and evaluation practice.
• Bayesian Online Changepoint Detection (Adams & MacKay): classical baseline.