Research letters · 6 November 2026
Research letter, 6 November 2026
This week
The most useful paper this week shows that watching shadow signals safely costs a lot of power. An anytime-valid promotion test cut false promotions from 94.5% to 0%, but it caught a 9.6%-a-year edge only 7.7% of the time. Two papers put LLM research agents on trial. Claude Opus 5.5 claimed predictability in 21% of worlds with no signal, and a one-line hint about "prior research" changed its verdicts. Separately, one LLM-proposed rule had Sharpe near 3 in discovery and validation and 0.26 on its single holdout look. The rest is about regimes: a "sticky" HMM follow-up to last week's paper that was tuned on its own test window, and a calibration result that should change how we score anything regime-dependent. All sources are arXiv. We could not verify any new SSRN or NBER papers from the past two weeks.
1. Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents (PROMOTE)
Benhamou, Ohana, Guez, Setrouk, Saadi, Jacquot · 2 Nov 2026 · ICAIF 2026 · arXiv:2611.03384 · no public code (seeds and a hypothesis ledger are described as "accompanying the artifact")
Method.
- Score: each period's net-of-cost return is divided by a scale fixed in advance and clipped to [−1, 1]. The scale is the larger of 50 bp and the 99.5th percentile of absolute net return in the training window.
- E-process: a running product of (1 + bet × score). It averages five fixed bet sizes {.025, .05, .10, .20, .35}, so the best bet can't be picked afterwards.
- Testing budget: each candidate rule gets a share π of the budget when it is created. The shares of every rule ever created sum to at most 1. Abandoning a rule doesn't refund its share, and changing a parameter means a new rule with a new share.
- Promotion: a rule is promoted when its e-value reaches 1/(α·π). Ville's inequality bounds the chance of ever promoting a rule with no edge, under unlimited peeking, rules created at data-dependent times, and dependent rules.
What they tested and found.
- Data: 24 calendar-rule hypotheses on three panels (6 equity ETFs, 5 bond/commodity ETFs, the 10 Fama–French industries). Rule direction was fixed on 2006–2010, and the test runs Jan 2015 – May 2026. Costs 5 bp per dollar traded.
- False promotions: across 3,000 sign-flipped null paths, repeated p < .05 tests promoted something on 94.5% of paths. A per-rule threshold did so on 0.7%, and PROMOTE on 0.0% (upper bound 0.13%).
- Agent-driven null: an agent chose which rules to create and when, from their past scores. False promotions were 93.1% for naive tests, 18.4% for per-rule thresholds and 1.77% for PROMOTE.
- Power is the cost. Injected edges, as annualized gross alpha: 4.8%, never detected; 9.6%, detected 7.7% of the time; 14.4%, detected 70%, with a median crossing at session 2,327 (about 9 years); 19.2%, 99%.
- Real-data replay: 0 of 24 rules promoted, and the best e-value was 1.93 against a bar of 480. The naive gate promoted a reversed January rule in 2016. That rule lost 0.9% a year over the full period.
Limitations.
- Power at realistic effect sizes is close to zero. Most cross-sectional equity edges are well under 10% a year net. On this evidence, PROMOTE would almost never promote one.
- Unused error budget: the observed false-promotion rate is far below α, so power is being left on the table. A sharper bet sizing could likely recover some of it.
- Weak test library: only 2 of the 24 rules had a positive mean. The zero promotions on real data show safety, not that the test can find a real edge.
- Decaying edges: evidence never expires, so an edge that fades is slow to cross.
- Unexplained parameters: the bet grid and the 50 bp floor aren't justified.
Evidence vs speculation.
- EVIDENCE: repeated unadjusted looks at a family of shadow rules will almost surely promote noise; PROMOTE holds false promotions near 0–2%, including when an agent chooses rules adaptively. This is simulation plus replay, so null-calibrated rather than market out-of-sample.
- EVIDENCE: power is very low below about 15% annual alpha (simulation).
- SPECULATION: that this is practical as a production gate for realistic edges. The paper's own numbers argue against it.
What we will do with it. Our daily shadow comparisons and short observation checklist are exactly the "watch until it crosses" pattern this paper models; our pre-registered one-shot studies don't have the problem, the ongoing watch does. We will replay PROMOTE on the 112 settled weekly books from our July regime-conditioning study, one stream per policy, with the paper's bet grid and scale rule fixed before running, and measure its power at our own effect size of about +0.15% a week by injecting that edge into sign-flipped nulls. If the median simulated time to promotion at that edge exceeds 3 years, PROMOTE is dropped as a candidate gate and only its bookkeeping is adopted: every shadow variant gets a ledger entry at creation, and nothing is refunded. Either way, no production change.
2. Evaluating LLM Research Agents as Tests of Return Predictability
Chang Liu · 2 Nov 2026, revised 8 Nov · arXiv:2611.04070 · replication package "planned"; available on request. (The first version carried the title "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds".)
Method.
- Worlds: simulated returns r = μ + βx + ε, with β set so the population R² is 0, 0.25, 0.5, 1, 2 or 4%. Each world hides the true predictor among 8 shuffled candidates; the other 7 are AR(1) distractors.
- Data seen: the agent gets 700 training months; a 360-month test window stays sealed. Data are anonymized, including random sign flips.
- Task: build a model that beats the historical mean, with a budget of 40 tool calls. The agent returns beats / does not beat / abort, plus a probability.
- Models: Claude Opus 5.5, GPT-6.1 Sol and Terra, and Qwen3-32B.
- Comparison: each agent's claim rate is compared with fixed statistical tests (Clark–West, HAC with Holm correction, IVX) at a matched false-positive rate. There are also "trap" null worlds (persistent predictor, GARCH, a regime world) and a framing manipulation that adds one sentence about prior research.
What they tested and found.
- Opus, no-signal worlds: claimed predictability in 21% of them (15% in linear nulls, 30% in persistent nulls).
- Opus, weak signal: at R² = 1% it detected 61%, versus 59% for HAC-Holm at the same false-positive rate. The difference is +2 points (95% CI −16 to +23), so the agent is no better than a textbook test.
- Calibration: Opus's stated probabilities had slope 0.36 and AUC 0.65.
- Framing: claim rates were 32 / 46 / 61% under skeptical / neutral / optimistic framing. One optimistic sentence doubled false claims, from 15% to 32%.
- Gaussian worlds: false-claim rates were Opus 34%, Sol 16%, Terra 44%, Qwen 35%, against 8% for HAC-Holm. No agent beat the matched test. Qwen claimed signal almost as often at R² = 0 (40%) as at R² = 1% (44%).
- Sealed-holdout test: keeps false claims at or below 5%, but detects only 25–28% at R² = 1%.
Limitations.
- Thin anchor: the matched false-positive rate rests on 40 null worlds (95% CI 7–29%).
- Contaminated studies: a bootstrap artifact let agents look up repeated shocks in Studies 1–2. It was found afterwards and partly corrected.
- Simplified setup: single-predictor worlds, one scaffold and one brief. Real research has many more garden paths.
- Model drift: model versions change. The direction of the results is the durable part.
Evidence vs speculation.
- EVIDENCE: frontier agents over-claim predictability in null worlds, and their verdicts move with one sentence of framing (pre-registered via tagged commits; sealed test window).
- EVIDENCE: at matched false-positive rates, agents don't beat a fixed HAC/Holm test.
- SPECULATION: how large the over-claiming is in our setting, where the agent sees real data, many candidates, and a human who wants a result.
What we will do with it. Much of our signal research is done with AI agent sessions. Our past kills worked because the decision rule sat outside the agent: matched nulls and placebos fixed in advance. This paper says to keep it that way, and to watch the wording of briefs. We will run a planted-signal audit of our own workflow: 20 synthetic worlds built on our real weekly price panel with returns permuted within each date, 10 pure null and 10 with one planted characteristic at a weekly rank IC of 0.02, each given to a fresh agent session under our standard brief, once with neutral wording and once with "early results look promising". If the agent's false-claim rate in null worlds exceeds 10%, or framing shifts the claim rate by more than 15 points, agent verdicts become advisory only and every promotion memo must cite a fixed external test the agent didn't choose.
3. Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found
Henry Zhang · 4 Nov 2026 · arXiv:2611.06607 · code: github.com/henryzhangpku/autonomous-quant-researcher
Method.
- Proposals: the LLM proposes strategies as declarative JSON from a fixed vocabulary. It never writes code; trusted code evaluates everything.
- Data splits: discovery, validation, and a holdout that can be opened exactly once, after an irreversible receipt is written.
- Gates, all of which must pass: positive net edge and P&L after costs; a positive lower bound from a bootstrap clustered by week; still positive at higher costs; no losing year in validation; minimum numbers of trades and weeks.
- Bookkeeping: near-duplicate proposals are refused before they use up a trial. Every proposal, refusal and crash goes into a hash-chained, append-only ledger.
What they tested and found.
- Campaigns: about two dozen, covering 0DTE options priced from traded bars, overnight SPX spreads, daily ETF rules, Kalshi binaries, earnings options and off-exchange volume.
- QQQ 0DTE survivor: Sharpe 2.94 in discovery (221 trades) and 3.34 in validation (43 trades), then 0.26 on its single holdout look (72 trades). Net edge fell from about +5% of spread width to +0.5%. It passed 8 of 9 holdout gates and was rejected.
- SPY 0DTE put credit spread: +56–72 bp of width per session with a 95% win rate. It was not promoted, because 13–30% of the payoff was market drift and the drift-neutral bootstrap bound was slightly negative.
- Other campaigns: 232 daily-rule cells gave 4 discovery passes and 0 promotions. 138 Kalshi hypotheses gave 0 (one look-ahead artifact was caught). 120 trade-management rules gave 0.
- Two survivors, both reported thinly: off-exchange volume share has a negative IC, t between −3.1 and −5.8, negative every year 2021–2026; and losers revert on market-wide crash days.
Limitations.
- The survivors come with t-stats only. No Sharpe, costs or holdout numbers are given; they are deferred to future papers.
- No multiple-testing control across campaigns. The 5% bootstrap tail is split only across a shortlist of at most 3.
- Gate behaviour never measured. There is no simulation of the gates' false-positive rate or power, so "hard to fool" is argued, not measured. That is the gap item 1 fills.
- Imperfect pre-registration: several campaigns predate the current schema, and one specification was lost and rebuilt before its holdout was opened.
- Short 0DTE windows with no 2022-style bear market.
Evidence vs speculation.
- EVIDENCE: one LLM-proposed rule that passed two stages collapsed on a true single-use holdout (Sharpe 3.34 → 0.26).
- EVIDENCE: drift accounted for a large share of an apparently strong short-premium payoff.
- SPECULATION: that the two survivors are real. The data needed to judge them isn't in this paper.
What we will do with it. Two ideas are directly usable: the holdout that can be opened only once, with a receipt, and the drift-neutral bootstrap. Our "net excess vs SPY" already removes drift for long-only books; any short-premium or market-timing idea would need the drift-neutral check explicitly. We will also run a trial-count audit from our own research records: reconstruct how many variants were actually tried for each signal family that currently survives in shadow, and recompute each survivor's deflated Sharpe with that count. Any survivor whose deflated Sharpe drops below 0.95 is downgraded to exploratory and needs a fresh one-shot holdout before promotion is discussed.
4. Mitigating Return Drag and Regime Whipsaws in Sector Rotation: An Asymmetric Downside-Aware Nested Clustered Optimization Framework
Supriadi · 3 Nov 2026 · arXiv:2611.05310 · code: github.com/obo-cloud/sp500-asymmetric-nco
This is a new paper, not a revision, by the author of the 30 October letter's item 1 (2610.27751). There, regime churn destroyed the regime-conditioned portfolio. This paper adds the persistence fix that was missing.
Method.
- Universe: the 11 SPDR sector ETFs.
- Risk model: downside semi-covariance (Bawa–Lindenberg) replaces covariance, both in clustering and in allocation.
- Regime model: a 2-state HMM on monthly SPY return and standardized VIX. Persistence comes from a "sticky" prior that strengthens self-transitions (κ = 0.30), and the regime changes only when the posterior is at least δ = 0.65.
- Allocation: in the bull state, maximize Sortino; in the bear state, minimize downside variance.
- Walk-forward: rolling 24-month window, monthly rebalancing.
What they tested and found.
- Out of sample: Aug 2020 – Dec 2025, 65 months. The main tables assume 0 bp costs.
- Proposed model: CAGR 13.41%, Sharpe 0.63, max drawdown −14.2%, Calmar 0.94, turnover 30.5% a month, 4 regime switches.
- Baseline regime-NCO: 11.05%, Sharpe 0.52, max drawdown −21.8%, turnover 60.8% a month, 16 switches.
- Plain NCO: 10.65%, Sharpe 0.58, turnover 27.0% a month.
- SPY: 16.36%, Sharpe 0.86, max drawdown −23.9%. Equal-weight sectors: 14.28%, Sharpe 0.78, max drawdown −18.3%.
- At 20 bp costs: proposed 12.60%, plain NCO 9.94%.
- Overfitting checks: CSCV PBO = 0.85. Deflated Sharpe is 0.998 for the proposed model, but 0.9999 for SPY.
Limitations.
- Tuned on the test window. κ and δ were checked on a 3×3 grid that re-ran the whole 65-month out-of-sample period. The chosen point sits next to a cliff: κ = 0.3, δ = 0.60 gives 11.5% CAGR and −19.2% max drawdown. With PBO = 0.85, the author concedes that selection by in-sample rank is unreliable.
- Loses to the plain benchmarks. It loses to SPY and to equal-weight sectors on CAGR, Sharpe and Sortino. It wins only on drawdown and Calmar, and that mostly reflects one episode (2022) with 4 switches in total.
- Misleading turnover headline. "Turnover halved" is measured against the churny baseline. Turnover is higher than plain NCO's.
- Confounded comparison. The baseline differs in turnover penalty (0.010 vs 0.015) and HMM inputs, and there's no variant with the sticky HMM but ordinary covariance.
- Implausible significance figures. The PSR/DSR values near 1 for a Sharpe of 0.5–0.6 over 5.4 years look miscomputed; a rough t-stat is about 1.2.
Evidence vs speculation.
- EVIDENCE (weak): a persistence prior plus a posterior threshold cut switches from 16 to 4 in this sample. The mechanism is plausible and matches the earlier diagnosis.
- SPECULATION: that it improves risk-adjusted returns. The paper's own benchmarks and PBO say no.
What we will do with it. This strengthens the hysteresis experiment proposed on 30 October and adds a warning: choose the persistence setting before looking, once. Our July study found the regime layer works as a light-touch controller (cash only in stressed or severe states), so a dwell rule would mainly change how often we enter and leave cash. We will pre-register exactly one hysteresis rule for the regime labels that controller uses, enter stressed at the current threshold and exit only after two consecutive weekly formations out of stressed, with no grid, and replay it on the same 112 books. If 20-day net excess vs SPY is not at least as good as the version without hysteresis, or label flips don't fall by at least 30%, hysteresis is dropped and the classifier stays as it is. No second dwell length gets tried on this window.
5. Aggregate Calibration Can Mask Stress-State Undercoverage: Bounded-Memory Forecast Intervals under Recurrent Financial Stress
Ruihang Han, Li-Hsiang Lin · 3 Nov 2026 · arXiv:2611.04223 · code on request
Method.
- Forecast: next-day log high–low range from a HAR model, with conformal intervals (target 90% coverage) and a memory budget of 250 past errors.
- Memory designs compared: most-recent-only (FIFO); "Rare replay", a recent queue plus reservoirs of past errors kept separately for each VIX state, with only the current state's reservoir used; matched controls; and an unbounded-history reference.
- VIX states: lagged VIX below 20, 20–30, and 30 or more, fixed in advance.
- Samples: an 11-ETF panel and a 50-stock panel. Evaluation 2012–2025, 3,520 dates, of which 156 are high-VIX.
What they tested and found.
- Recent-only memory: 90.0% coverage overall, but only 75.8% on high-VIX dates in the ETF panel and 77.6% in the stock panel. The gaps vs other dates are −14.9 pts (95% CI −27.8 to −3.8) and −13.0 pts (CI −24.4 to −3.2).
- Rare replay: high-VIX coverage 84.8% / 86.2%. Unbounded history: 90.1% / 90.9%.
- Interval score: Rare replay beats recent-only on high-VIX dates in both panels, by small margins: CIs end at −0.0018 and −0.0009. The gain is largest when stress returns after a quiet spell.
Limitations.
- Few, shared episodes: only 13 stress episodes, on one calendar common to both panels, so the panels are not independent replications.
- Fragile significance: several CIs barely exclude zero with no multiplicity adjustment, and some cross zero under an alternative stress definition.
- Survivorship: the stock panel isn't survivorship-free, as the authors acknowledge.
Evidence vs speculation.
- EVIDENCE: aggregate calibration hides about a 13–15 point coverage shortfall in stress states, out of sample, in two panels.
- SPECULATION (thin): that state-structured memory reliably fixes it.
What we will do with it. Our regime classifier standardizes breadth and concentration over a 63-day window, which is a bounded, recent-only memory. When stress returns after a calm stretch, the window no longer contains the last stress episode, so the scores may understate how unusual today is. That is the paper's failure mode. We will recompute the classifier's combined score two ways on the existing history, the current 63-day window and an expanding window, and compare the label each produces at the start of the eight stressed or severe episodes in the July study. If the expanding window flags stress earlier in fewer than three of them, we close the question and keep 63 days; if three or more, we open a pre-registered controller replay with no threshold changes before then. One caution from a paper in Watching (2610.40366): with only eight stress weeks, any comparison on stress dates is a sample of eight. Treat the result as a description, not a test.
6. PanelGP: A Probabilistic Factor Model for Asset-Return Panels
Andrea Ruglioni, Damir Filipović · 2 Nov 2026 · arXiv:2611.04040 · code: github.com/Ruglio/PanelGP (data need WRDS)
Method.
- Model: a conditional factor model. Firm characteristics determine factor loadings and macro conditions determine expected factors. Both relations get Gaussian-process priors, and the kernel choice nests IPCA as the linear case.
- Inputs: 137 rank-mapped characteristics and 6 context variables.
- Estimation: sparse variational inference. One fit gives both expected returns and a covariance matrix in closed form.
What they tested and found.
- Universe and sample: historical S&P 500 members using CRSP membership dates, so no current-constituent survivorship. Test period 1995–2024. In the rolling design, the model is re-estimated each year on the latest 360 months.
- Net Sharpe at 10 bp: tangency portfolio 1.52, vs 1.35 for IPCA with an intercept and 0.85 for plain IPCA. Long-short decile: 0.91 vs 0.72 vs 0.36. The tangency portfolio breaks even at 49.7 bp.
- Ranking vs R²: rank correlation is 4.84% vs 3.69% for IPCA with an intercept, but rolling R² is worse: 0.20% vs 0.64%.
- Risk: minimum-variance volatility 12.3% vs 13.2%.
Limitations.
- Not significant. The headline Sharpe gain over IPCA isn't statistically significant; only the log-score and minimum-variance differences are. Several figures here come from a summary of the paper body rather than a line-by-line reading.
- Halves after 2010: tangency Sharpe falls from 2.62 (1995–2009) to 1.16 (2010–2024), the familiar pattern of anomaly decay.
- Light costs: 10 bp with no borrow cost or impact is generous for a tangency portfolio with short positions.
- Model search: 4 kernel pairs × 8 factor counts × 5 seeds in the fixed design. Model selection used a separate validation period, which helps.
Evidence vs speculation.
- EVIDENCE: nonlinear characteristic maps rank stocks better than linear ones over 30 years out of sample on survivorship-free S&P 500 data (rank IC, log score).
- SPECULATION: that the net Sharpe edge over IPCA is real, and that it survives realistic costs after 2010.
What we will do with it. The transferable finding is the split result: better ranking with worse R². Our scorer is a ranker, so rank IC is the right yardstick, and our fundamentals are neutralized, so we can't replicate a 137-characteristic model anyway. We will compute the weekly cross-sectional rank IC of our four momentum components individually and as the production linear blend, on replay history before 2024 that was never used for tuning, and add exactly one fixed nonlinearity, the rank of the component product. If its incremental IC over the linear blend is under 0.5 percentage points, or flips sign in more than one regime block, it is dropped, with no search over other interactions. We rank this last because the paper's own significance is weak.
Promising experiments
- Planted-signal audit of our own agent research (item 2). It's cheap, needs only synthetic worlds on real panels, and directly tests the process that produces most of our verdicts. The framing arm matters as much as the null arm.
- PROMOTE replay plus power check on the 112 books (item 1). Even if PROMOTE is too weak as a gate, it tells us how long our shadow watch would need to run to separate +0.15% a week from noise. That number should set observation windows, not a 5-day checklist.
- Honest trial-count deflated Sharpe (item 3). It's an afternoon of reading our own records and has no new-data risk. Run it before either regime experiment (items 4 and 5), since it may already downgrade the cash-in-stress policy.
Watching / not yet credible
- Comparing tail forecasts (2610.40366): the effective sample size of a tail-forecast comparison is the number of exceedances, and Diebold–Mariano favours the less conservative forecaster when exceedances are scarce. In 250-day windows across 7 indices, 18.4% of significant DM results contradicted the ranking on the remaining days. This is method, not alpha, and it matters for any stress-state claim we make.
- Varner, interpretable allocation and MPC rebalancing (2611.03022): 373 US equities, traded Jan 2025 – Sep 2026, with code. Score-based picks per sector did no better than random, and the hand-picked list won. On synthetic markets, the generator's built-in trend persistence flattered score-based selection. That is a useful warning for any synthetic test of a momentum scorer. The sample is survivorship-biased (admitted), about 20 months long, and has no significance tests.
- Safe portfolio learning from untrusted predictions (2611.04826): honest negative. The pre-specified signal orientation failed on a frozen 82-stock holdout, and flipping its sign afterwards didn't help either. The controller kept capital within 16 bp of the safe benchmark. The authors claim safety, not alpha.
- Collision-aware layer (2611.03453): penalizing exposure to the recent common return mode cut max drawdown from 23.6% to 14.1% on 10 ETFs, 2012–2026, at 10 bp. The 95% CI on the improvement is [−3.3, 18.4] points. Not yet credible.
- LENS (2611.04217) and FRPO (2611.07750): LLM-graph and RL stock prediction on S&P 500 / DJIA. They report gains over baselines, but the abstracts mention no costs or held-out era.
- Mosaics of Predictability (NBER WP 35158, Cong, Feng, He, Wang) reports out-of-sample Sharpe around 2, with predictability concentrated in large-surprise, low-volume stocks. We could not confirm its posting date or read it this week, so it isn't covered.
Updates
Regime-NCO (2610.27751, covered 30 October) has a sequel, item 4 above, rather than a revision. The sequel's PBO of 0.85 reinforces the earlier reading that the regime overlay's problems were about selection as well as churn.
Nothing previously covered was materially revised.
This letter is drafted with AI research assistance and reviewed and signed by Bob Katz. It is research commentary, not investment advice. No result described here is a claim net of trading costs unless stated, and the experiments listed are planned tests of our own models, not findings.