Anatomy of self-deception: five bugs that manufactured alpha

Abstract

The dangerous failure mode in quantitative research is not the number that looks wrong — it is the number that looks right. A Sharpe ratio of +1.5 gets scrutinised; a Sharpe of +7.4 gets celebrated for about six hours and then someone asks why the compound return of the same strategy is −100%. We document five bugs from our own program, in chronological order of discovery, each of which produced an internally consistent metric that was not what its name claimed. The common root cause across all five: the code computed a number correctly, but not the number. The common fix: mechanical, pre-publication plausibility contracts that treat every headline figure as guilty until proven consistent with its own diagnostics.

Bug 1 · Future-leaking centring: one line, four Sharpe points

Our specialist strategies trade the sign of a model's prediction relative to a rolling median — the “centring” step. An early implementation computed that median over a window that included the bar being scored. The result: a strategy whose backtest Sharpe was +2, and whose corrected, causal version was −2. One line of code, a four-point swing, and the broken version was the profitable-looking one.

What made this insidious is that the leak is invisible at the trade level. Each individual trade looks legitimate; only the aggregate is impossible. The full case study, including the corrected centring contract (rolling median over trailing 336 hours with a 168-hour warmup, initialised from the training window) is in the honest-centring paper.

Bug 2 · Rank targets passed as returns: +7.38 becomes −3.53

In May 2026 an evaluator reported that a K-of-N dollar-neutral portfolio built from our LGBM ranker earned +7.38 annualised Sharpe net of cost. The model had been trained with target_mode=xs_rank — its targets array contained cross-sectional ranks scaled to [−1, +1], not forward returns. The evaluator multiplied positions by those ranks and called the result dollar PnL. It was actually a rank-IR — a fine screening statistic, but not money.

Rerun on raw forward returns at the live cost convention, the same portfolio scored −3.53. The full post-mortem is in IC is not Sharpe. What is worth repeating here is how it was caught: not by the code, but by a human noticing that a +7.38 SR strategy with a −100% compound return is a logical impossibility. The number was internally consistent and externally absurd.

Bug 3 · h-bar overlap in the cost function: the +8 SR overlay

A diagnostic study concluded that adding a per-asset stop-loss overlay to our specialists added +8 Sharpe. The overlay function evaluated stops using PnL paths that overlapped the forward-return horizon — with a 48-hour holding period, adjacent hourly bars share up to 47 hours of the same forward window, so the “stop” was effectively reading 47 hours into the future and exiting before losses it had already seen.

The corrected study, using strictly non-overlapping trade windows, found the overlay added nothing. The +8 was pure look-ahead, laundered through an execution overlay rather than the signal itself — which is why the signal-level leakage checks did not fire.

Bug 4 · The mislabelled panel: 215 files, wrong coins

A per-coin attribution pipeline mapped asset IDs to coin names through a lookup (_aid_to_coin_map) that was built in a different order than the panel it described. Every per-coin IC in a 215-file result set was attributed to the wrong coin. The aggregate numbers were correct — the pooled IC does not care which coin is which — so nothing looked wrong until per-coin decisions (which names to keep, which specialists to promote) started disagreeing with independent evidence.

All 215 JSON/MD artefacts were relabelled in place and the ClickHouse heartbeat rows re-pushed. The lesson is narrower than the others but general: label integrity is part of the metric. A correct number attached to the wrong entity is a wrong number.

Bug 5 · Non-reproducible training: results that vanish on rerun

In late May 2026 we discovered that rerunning the same training cell with the same seed did not produce the same model. Un-seeded data loader shuffling, non-deterministic GPU kernels, and thread-count-dependent LightGBM histograms meant our “seed ensembles” were partly sampling build noise. Every result produced before the determinism patch has an unquantifiable error bar on top of its stated one.

The fix was a determinism patch validated by a twice-run gate: eight representative architectures retrained twice from identical seeds must produce byte-identical predictions before any run counts. The entire 30-architecture grid was then retrained from scratch under a new run ID. Expensive — roughly a full GPU-week — and non-negotiable: a result you cannot reproduce is an anecdote.

The meta-bug: selection on noise

Behind all five sits the bug that no single check catches: run enough variants and the best backtest is an order statistic, not an estimate. A June 2026 audit of our own earlier promotion decisions found that promoted specialists' backtest Sharpes were statistically indistinguishable from what selecting the max of that many noisy trials would produce — and their subsequent paper performance regressed accordingly. That audit is why the promotion funnel described in the Grid A/B/C paper exists: pre-registered gates, seed medians instead of best seeds, and a hard rule that construction choices are settled by paper trading, not by further backtest iteration.

Scorecard

Two of five caught by a human before any automated flag existed. The plausibility contracts were written specifically to drive that number to zero going forward — the flags (EXTREME_SR, INCONSISTENT, OVER_CAPITAL, HEAVY_TAIL, UNUSUALLY_CLEAN) encode exactly the reasoning the human used, so the machine asks the questions first. How those contracts are enforced day-to-day is the subject of the research OS paper.