Methodology

The model

A 3-state Gaussian Hidden Markov Model with a full covariance matrix per state, trained via the Baum-Welch (Expectation-Maximization) algorithm using hmmlearn's GaussianHMM implementation. Hyperparameters:

• n_components=3 — three hidden regimes
• covariance_type="full" — each state has its own full covariance matrix
• n_iter=200, tol=1e-4 — convergence criteria
• random_state=42 — for reproducibility; same data + same seed always produces an identical model

Three states is the right count for a regime indicator: more states tend to overfit on daily data (the model finds spurious sub-regimes that don't generalize), and fewer collapses the meaningful bull/bear distinction.

Features (observations)

Each trading day is represented by up to four features. Features are z-scored (standardized to mean=0, std=1) before feeding into the HMM so no single feature dominates.

KOSPI and Shanghai use 3 features (no VIX). For those indices, realized_vol carries the vol signal alone — and is more than sufficient: see the regime tables on the dashboard.

State labeling (the bear/neutral/bull rule)

The HMM returns three states numbered 0, 1, 2 — but the numbering is arbitrary and can flip between training runs. To make labels stable, after each fit we compute a composite score for each state:

score[i] = -z(mean_realized_vol[i]) + 0.25 * z(mean_log_return[i])

Where z(.) is z-scoring across the three states. States are then sorted by score:

• lowest score → bear (highest vol, with low return as a tiebreaker)
• middle score → neutral
• highest score → bull (lowest vol, with high return as a tiebreaker)

Why vol-dominant? Returns alone barely separate regimes in long-bull markets like India — all three states tend to have positive mean returns. Volatility is the actionable axis: when vol is high, be defensive, regardless of price direction. The 0.25 weight on returns acts as a tiebreaker for ambiguous cases.

Filtered vs smoothed probabilities

The single most important methodological choice in a live dashboard is using filtered probabilities, not smoothed ones.

• Filtered: P(state[t] | observations[1..t]). At each day t, the model uses only data up through that day. This is what you'd actually have if you ran the model in real time.
• Smoothed: P(state[t] | observations[1..T]). The forward-backward algorithm uses future observations to refine past probabilities. Looks great in backtests, but cheats — you can't have it live.

This dashboard always shows filtered probabilities. We compute them via a direct forward-pass implementation of the log-alpha recursion:

log_alpha[t, j] = log_emission[t, j]
                + logsumexp_i( log_alpha[t-1, i] + log_transmat[i, j] )

filtered[t] = exp(log_alpha[t] - logsumexp(log_alpha[t]))

This means: today's probabilities will be revised tomorrow as new data arrives, but past displayed probabilities are never revised retroactively using future data.

Data sources and dates

Daily OHLC data is pulled from Yahoo Finance via the yfinance library:

All indices have ~4,000 trading days of data. Yahoo Finance is a free, unofficial data source — adequate for personal research, but commercial deployments should use a paid feed (e.g., EOD Historical Data, Alpha Vantage). The DataSource abstraction in the code makes the swap a one-file change.

Update cadence

What we deliberately don't do

• No trading-strategy performance numbers. The model is published as a live regime indicator, not a backtested trading system. Binary timing strategies on equity indices typically underperform buy-and-hold for any signal — we keep the focus on the regime read itself, not on simulated returns.
• No trade signals. No "buy" or "sell" outputs. Only regime probabilities.
• No leverage or position sizing recommendations. That's your decision.
• No auto-execution. The system has no broker integration and is not designed to.
• No smoothed probabilities in the live view. Only filtered. The smoothed pass exists in the codebase for offline historical analysis only.

Honest limitations

• HMM lag: filtered probabilities will lag a regime change by 1–3 days. This is fundamental to the math — to detect that the regime has changed, the model needs to observe some post-change data.
• State labels assume regime persistence: the transition matrix is sticky (~0.95+ diagonals). If real markets have rapid regime switches, the model will be slow.
• Three states is a simplification. Real markets have more nuance (low-vol bull rally, high-vol bull rally, panic, slow grind, V-shape recovery, etc.). The 3-state quantization is a deliberate tradeoff for interpretability.
• Features are the model's universe. If something matters (e.g., bond yields, commodity prices, credit spreads, sentiment), but isn't in the feature set, the model is blind to it.
• Past patterns may not generalize. The model assumes the statistical structure of returns is stationary across the training window. Big structural changes (new asset classes, regime shifts) can break this assumption.

Verification

A comprehensive audit was performed across seven layers — raw data accuracy, alignment and gaps, feature math, HMM training internals, label permutation, filtered probability correctness, and database parity. All 100+ assertions passed; the math is fully reproducible and the probabilities are provably causal (verified by truncating data and confirming byte-identical past values).

Known data-quality concerns: 4 days of suspect USD/INR ticks in Jan-Feb 2012 and 2 days of suspect USD/CNY ticks in Jul 2011 from Yahoo Finance. Impact is bounded to ~2 weeks of localized regime classification near those dates and does not affect any high-confidence historical calls (COVID, 2017–18 bull, 2022 bear, etc.).