Detecting data drifts in dynamical systems

A structural-change detector that works zero-shot on series it has never seen: 42 statistical features distilled from a 170+ candidate pool feed a 3-2-1 stack of heterogeneous learners (TabPFN, XGBoost, Random Forest, CatBoost, LightGBM), fused interpretably with RuleFit and wrapped in a POMDP layer that turns pointwise scores into a regime belief.

Every changepoint detector, from CUSUM and Page-Hinkley to a classifier trained on labeled breaks, quietly learns the idiosyncrasies of the series it was tuned on. It learns that "volatility above X means trouble" on the data where that happened to be true; then the regime shifts, the shortcut stops mapping to reality, and the detector is at its most confident exactly when it should be at its least. That failure mode turns the problem inside out: a detector meant to survive regime change cannot be allowed to learn any single domain's quirks. It has to learn what a break looks like, an invariant signature in the statistics of the signal, and nothing else.

I'd been calibrating discretionary tools against single-domain models, an XGBoost here, a random forest there, and it was fine until it wasn't. Each model was accurate on the distribution it was trained on and ambiguous the moment the generating process moved, which is precisely the moment I needed a trustworthy answer. Retraining per regime is circular: you need to detect the regime change to know you should retrain.

The push to build something serious came from the ADIA Lab Structural Break Challenge, and specifically from the write-up by the winning Alphabot team, who took first place with a stack of diverse first-level tree models merged by a meta-model. Their result validated the core intuition: no single tree architecture is unbiased, but their biases are different, and stacking distinct statistical perspectives beats tuning any one of them. What I wanted to add on top of that recipe was threefold: a deliberately heterogeneous training corpus (financial market data, synthetic assets built by delta replication, 12-lead ECG recordings, weather and energy time series) so the model is applied zero-shot to any new series without retraining; an interpretable fusion via RuleFit instead of an opaque meta-model; and a sequential decision layer, a POMDP, that turns pointwise scores into a belief about the current regime. The output is a calibrated structural-change score used upstream of decision tools as a confidence throttle, but everything below is about the machine, not the use case.

01How it works

The system is four stages: a feature distillery, a 3-2-1 stacked ensemble, a RuleFit fusion layer, and a POMDP temporal layer. Data flows as: raw series -> causal rolling windows -> 42-dimensional feature vector per window -> layer-1 learners (TabPFN, XGBoost, Random Forest) -> layer-2 meta-learners (CatBoost, LightGBM) -> layer-3 reducer (single shallow gradient-boosted tree) -> rule extraction -> sparse linear rule model -> per-window score -> POMDP belief update -> regime belief and throttle action.

Feature distillery: 170+ -> 42

Each window is summarized by statistical descriptors drawn from a candidate pool of 170+, organized in nine families:

1. Distributional: moments, quantile spreads, tail indices, mode multiplicity.
2. Autocorrelation / linear structure: ACF and PACF summaries, AR(p) fit coefficients and residual diagnostics.
3. Spectral: band energies, spectral centroid and flatness, spectral entropy.
4. Complexity / entropy: permutation entropy, sample entropy, Lempel-Ziv complexity.
5. Recurrence: recurrence quantification: recurrence rate, determinism, laminarity, trapping time.
6. Scaling: Hurst-type exponents via DFA and R/S, multifractal width.
7. Stationarity statistics: rolling-split test statistics (mean, variance, distributional distance between window halves).
8. Wavelet: energy ratios across dyadic scales, wavelet leaders.
9. Model-comparison: likelihood gaps between simple competing generative fits (e.g. constant-variance vs. switching-variance), in the same spirit as catch22's "which model explains this best" features.

The pool is filtered along six axes. Each axis is a hard gate with an explicit test and pass criterion, not a soft weight:

Selection then proceeds in three passes. First, the hard gates above cut the raw pool. Second, stability selection: the probe-ensemble permutation ranking is recomputed across bootstrap resamples of the training domains, and only features selected in the large majority of runs stay, which kills features whose importance depends on which domains happen to be in the training split. Third, a greedy mRMR-style forward pass (maximize importance, penalize redundancy with the already-selected set) grows the final set, stopping when the marginal gain on held-out series spanning every domain drops below epsilon. Forty-two features survive.

Before entering the ensemble the 42 are residualized against slow trend, rank-normalized causally (statistics from past windows only), and whitened; this is the step that lets a barometric pressure series and an order-flow series share one feature space. Level and scale are deliberately destroyed; only shape and dynamics remain.

The 3-2-1 stack

Layer 1 holds three learners chosen for maximally decorrelated inductive biases, not for individual accuracy:

• TabPFN: a prior-fitted transformer that does Bayesian-flavored in-context learning on tabular data. Its prior over causal structures makes it strong on small, clean windows and gives it error modes unlike any tree. The 42-feature budget is not incidental: it keeps every window inside TabPFN's input constraints.
• XGBoost: sequential bias reduction; sharp axis-aligned splits, aggressive on interactions, level-wise growth.
• Random Forest: variance reduction by bagging; smooth, conservative, hard to destabilize.

Each layer-1 learner emits out-of-fold probabilities under a purged, embargoed blocked scheme (in the spirit of Advances in Financial Machine Learning). The non-obvious part is why purging is mandatory here: overlapping rolling windows mean a naive K-fold leaks, because a window ending at t shares raw samples with a window starting at t-k in another fold. So any training window whose span overlaps a test block (plus an embargo margin after it) is dropped before fitting, and layer 2 only ever sees these out-of-fold probabilities, never in-fold predictions.

Layer 2 is two meta-learners trained on the layer-1 probabilities concatenated with a compressed slice of the raw features; they are allowed to learn when to trust whom, conditional on signal statistics:

• CatBoost: ordered boosting and ordered target statistics make it the least leakage-prone gradient booster available, which matters more at the meta level than anywhere else: meta-features are model outputs, and target leakage there is amplified, not diluted.
• LightGBM: leaf-wise growth with histogram binning; it carves deep, narrow interaction paths that level-wise XGBoost tends not to find, keeping layer 2's two members decorrelated from each other as well as from layer 1.

Layer 3 is a single shallow gradient-boosted model that reduces the stack to one per-window score. The narrowing shape (3 -> 2 -> 1) is the point. Width at the bottom buys disagreement; each subsequent layer is an arbiter over the previous one's disagreements, and shallow capacity at the top forces the arbitration to be simple. Empirically, the layer-1 disagreement itself, the variance across the three probability streams, turned out to be one of the most informative meta-features: heterogeneous models diverge hardest precisely in the neighborhood of a structural break, before any of them is individually confident.

RuleFit fusion

Instead of shipping the stack as an opaque scorer, every tree in the ensemble is decomposed into its root-to-leaf paths, each path becoming a binary rule ("permutation-entropy delta > a AND determinism drop > b"). Following Friedman & Popescu's RuleFit, an L1-penalized linear model is then fit over the rule indicators plus winsorized linear terms of the 42 features. The lasso keeps a few dozen rules out of thousands; the fused detector is literally a weighted rule list.

This buys two things. First, interpretability with teeth: partial-dependence checks run against the rule list directly, and any rule whose support is dominated by a single domain is a red flag for shortcut learning; several were killed this way during development. Second, regularization: the L1 fusion acts as a global pruning pass over the entire stack, discarding split structure that only ever fired on one domain's quirks.

The POMDP temporal layer

A per-window score is still a pointwise judgment; regime detection is a sequential problem, and the regime itself is never directly observed, only its statistical shadow through the features. That is a partially observable Markov decision process by definition, and the quickest-change-detection literature has long framed Bayesian break detection exactly this way.

Concretely: the latent state space is a small set of regime states (stable, transitional, broken, plus break-type refinements); the observation is the fused RuleFit score, with emission densities per state estimated on training domains; the transition prior is sticky, encoding regime persistence. A Bayes filter updates the belief over states after every window. Actions are discrete confidence-throttle levels, and the reward trades detection delay against false-alarm cost, so the operating point is chosen by the policy, not by an ad hoc score threshold. With a state space this small the policy is solved offline on the discretized belief simplex. The practical payoff is twofold: isolated score spikes get absorbed by the belief instead of triggering, and sustained moderate scores accumulate into a detection even when no single window is decisive, which is precisely the phase-shift and observability-loss signature that pointwise thresholds miss.

Training for invariance: the corpus is the regularizer

The corpus spans five deliberately dissimilar domains: financial market data across instruments and frequencies; synthetic assets constructed by delta replication, which provide market-like series whose generating process is known and controllable; 12-lead ECG recordings; weather series; and energy time series. The point of the mix and of the ECG data above all, is that these domains share nothing at the level of units, scale, sampling regime, or surface statistics. A quasi-periodic physiological signal and an order-flow series have no common shortcut to exploit. Any split rule that keys on a market idiosyncrasy is actively penalized by the ECG and weather portions of the loss, and vice versa. The only structure that pays off everywhere is the thing all five domains genuinely have in common: what a shift in the underlying dynamics looks like once level and scale have been stripped away. In other words, the corpus does not tell the model what a structural break is; it removes every cheaper thing the model could learn instead, and lets it find its own representation of "the dynamics changed."

Labels combine synthetic changepoints (injected mean/variance/spectral shifts and parameter drifts in controlled generating processes, the same break taxonomy the ADIA Lab challenge data covered: changes in mean, variance, distributional shape, dependence structure, and tail behavior) with annotated real-world breaks in the style of the Turing Change Point Dataset; the delta-replicative synthetic assets are especially useful here, since breaks can be injected into a market-like process with exact ground truth. Validation is always on held-out series the model never trained on, drawn from every domain, with cross-domain consistency, not any single domain's score, as the selection criterion during development. The target classes are the invariant break signatures: level/variance breaks, phase shifts, observability loss, and inflation of aleatoric uncertainty.

02Trade-offs

03Validation, and why there are no numbers here

I am deliberately not publishing benchmark tables, score distributions, or comparative statistics: the detector runs upstream of live trading tools, and its measured operating characteristics are part of that edge. What I can share is the methodology, which is where the credibility of a domain-agnostic claim actually lives anyway.

All evaluation is performed on held-out series the model never trained on, drawn from every domain in the corpus, with the acid test being the non-financial ones. A detector that scores well on held-out market data proves little (that is the overfit trap from the intro); one that scores consistently on held-out ECG, weather, and energy series, using the exact same frozen model, is evidencing invariance. Performance is tracked as break/no-break discrimination on labeled windows plus the detection-delay versus false-alarm trade-off against classical online detectors (BOCPD, Page-Hinkley, ADWIN) and against each layer-1 learner run solo. Two internal findings shaped the design and are worth stating without numbers: the stack's advantage over its best single member is largest on the domains furthest from finance, and the metric that governed every design decision was worst-domain performance, not average performance. A domain-agnostic claim lives or dies on its minimum, not its mean.

04Where it went sideways

The psychic detector. The first end-to-end run produced held-out AUROC numbers that were too good, and too good is a bug. Feature normalization was the culprit: rank-normalizing each feature over the full series meant every pre-break window's percentile was computed against statistics that included the post-break distribution. The detector wasn't detecting breaks; it was reading them out of the normalizer. In a backtest this is invisible: the numbers just look great. The fix was strictly causal rolling normalization (expanding or trailing window, past data only), and a standing invariant test: recompute every feature on the series truncated at time t and assert bit-equality with the streaming value at t.

Redundancy that wasn't. Early feature screening clustered redundant features on the pooled cross-domain correlation matrix and kept one per cluster. Two spectral features that were ~0.97 correlated on market data (and thus collapsed to one) turned out to be nearly independent on quasi-periodic physiological signals, and the discarded one carried most of the phase-shift signal there. Held-out performance on the physiological data cratered while every other domain looked fine, which is exactly the signature cross-domain validation exists to catch. The fix is the per-domain-then-intersect redundancy screening described above: a pair is only "redundant" if it is redundant everywhere.

05What I'd do better next time

The ensemble architecture turned out not to be where the difficulty lived: the 3-2-1 stack, the RuleFit fusion, and the POMDP layer went in mostly as designed. The real constraint, the thing every failure traced back to, was the feature distillery and its evaluation protocol: causality of every transform, per-domain validity of every screening decision, leak-freedom of every meta-feature. The model learns whatever the pipeline lets it learn, and every shortcut it ever found was one the pipeline had left open.

The next version would push that insight further in two directions. First, treat layer-1 disagreement as a first-class signal rather than a lucky meta-feature: model the distribution of cross-learner divergence explicitly, since it front-runs the individual detectors near breaks, and feed it to the POMDP as a second observation channel. Second, close the loop between RuleFit and feature screening: rule-support-per-domain turned out to be the sharpest shortcut detector in the whole system, and it currently runs as a manual audit. Automating it, rejecting features whose derived rules concentrate on one domain during selection rather than after fusion, would turn the interpretability layer into an active regularizer. That is the piece of understanding I didn't have at the start: interpretability wasn't the deliverable, it was the debugger.

06Further reading

• ADIA Lab Structural Break Challenge: the competition that framed the problem: tens of thousands of labeled univariate series with heterogeneous break types.
• How far can we push the winning model of the ADIA Lab Structural Break Challenge? (Brandao et al.): the winning Alphabot team's write-up; the diverse-stack-plus-meta-model design that inspired this project.
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second: the prior-fitted network in layer 1; its input constraints shaped the 42-feature budget.
• CatBoost: unbiased boosting with categorical features: ordered boosting, the leakage-resistance argument for using it at the meta level.
• LightGBM: A Highly Efficient Gradient Boosting Decision Tree: leaf-wise growth, layer 2's second member.
• XGBoost: A Scalable Tree Boosting System: layer-1 boosting learner.
• Predictive Learning via Rule Ensembles (Friedman & Popescu): the RuleFit fusion.
• Quickest Change Detection (Veeravalli & Banerjee): the sequential-detection framing behind the POMDP layer.
• catch22: CAnonical Time-series CHaracteristics: the feature-distillation philosophy this pipeline extends toward break sensitivity.
• Permutation Entropy: A Natural Complexity Measure for Time Series (Bandt & Pompe): core dynamics feature.
• Recurrence Plots for the Analysis of Complex Systems (Marwan et al.): recurrence quantification features.
• An Evaluation of Change Point Detection Algorithms (van den Burg & Williams): real-world changepoint labels and evaluation practice.
• Bayesian Online Changepoint Detection (Adams & MacKay): classical baseline.