Detecting multivariate market regimes via clustering algorithms

Apply clustering algorithms for market regime detection at the portfolio level

In partnership with

Hi! Here's Iván with this week's exciting newsletter, brimming with insights and discoveries on building robust investment strategies and risk models using Machine Learning.

In this edition, I am presenting the following sections:

  • 🕹️ AI-Finance Insights: I summarize three must-read academic papers that mix cutting-edge ML/DL with Quant Finance:

    • Detecting multivariate market regimes via clustering algorithms

    • Temporal and Heterogeneous Graph Neural Network for Financial Time Series Prediction

    • Deep-learning models for forecasting financial risk premia and their interpretations

  • 💊 AI Essentials: The section on top AI & Quant Finance learning resources: Today, I'm excited to share a 1-hour video that explores how finance teams can significantly enhance their efficiency using AI and automation!

  • 🥐 Methodological Insights: In this edition, I recommend a paper that combines the well-known volatility timing strategy with ML volatility prediction models, which are simple but effective.

Today’s Sponsor

Make Your Money a Multitasker.

With Betterment's expert-built ETF portfolios, you’re automatically diversified across thousands of stocks and bonds at once. These expert-built portfolios are designed to help reduce risk, regardless of what’s happening in the market.

AI-Finance Insights

“Detecting multivariate market regimes via clustering algorithms”

In my opinion, clustering methods are the way to go for market regime detection. This paper provides another angle to apply clustering algorithms for market regime detection at the portfolio level (multivariate time series).

Here are my simple ideas from the paper 👇

➡ The paper introduces a novel, model-free algorithm that extends the "WK-means" algorithm to analyze multivariate time series for dimensions greater than one. This method adeptly identifies changes in market regimes and correlation structures between assets using either Wasserstein distances or Maximum Mean Discrepancies.

➡ Integrating this new algorithm with a two-step clustering process, the research showcases its effectiveness on both synthetic and real-world data (S&P data), providing reliable estimations of mean, variance, and correlations critical for strategic financial analysis.

➡ Utilizing the derived statistical measures, the paper applies Modern Portfolio Theory to develop profitable trading strategies for pairs of assets, illustrating the practical financial applications of these theoretical findings.

➡ The algorithm’s flexibility is highlighted through its capability to adapt between different statistical distances, enhancing its utility across various financial datasets and scenarios.

➡ Furthermore, the research contributes a method to determine the optimal number of clusters in univariate time series, which complements the primary algorithm and supports more refined strategy development in portfolio management.

“Temporal and Heterogeneous Graph Neural Network for Financial Time Series Prediction” 

This paper proposes the use of a temporal graph neural networks to consider not only the relationships among stocks but also the dynamics of those relationships with a temporal component.

Here are my simplified takeaways from the paper: 👇

➡ The paper introduces a Temporal and Heterogeneous Graph Neural Network-based (THGNN) approach for dynamically learning relationships among financial time series. This method constructs daily company relation graphs and utilizes a transformer encoder to enhance price movement predictions.

➡ The THGNN significantly outperforms traditional methods by adapting to dynamic changes in company relations and applying heterogeneous graph attention networks to optimize financial data embeddings.

➡ Extensive tests conducted on the US and Chinese stock markets demonstrate the superior performance of this approach, particularly in a real-world quantitative algorithm trading system where it dramatically improved accumulated portfolio returns.

➡ Additionally, the study proposes an attribute-driven graph attention network that innovatively captures attribute-sensitive influence spillovers among firms. This model shows enhanced performance over conventional models like GCN and LSTM, improving decision accuracy and area under the curve metrics substantially.

➡ The attribute-driven approach not only advances predictions in financial markets but also suggests potential applications in estimating implied volatilities and bulk futures, showcasing its ability to handle complex, attribute-mattered information propagation effectively.

“Deep-learning models for forecasting financial risk premia and their interpretations”

Original method of enhancing the forecasting ability of returns by splitting market risk premia into two separate models (time series and cross-sectional components).

This approach is not only applicable to market risk premia but also to other investment strategies.

Here are my takeaways: 👇

➡ The paper divides the forecasting of financial risk premia into two separate models—one for time series and another for cross-sectional analysis. This separation improves performance and addresses the non-stationarity in stock returns while effectively predicting cross-sectional return spreads.

➡ Incorporating skip connections into deep neural networks enables the training of deeper models, significantly boosting their predictive accuracy over traditional methods.

➡ The study extensively tests these models across various metrics including different capitalization stocks and time intervals, and by evaluating ML-based portfolios, confirming their superior performance.

➡ Efforts to interpret these "black box" ML models using local approximation techniques provide clarity on the driving features of predictions and the relationships between inputs and outputs.

➡ The methodologies not only enhance risk premia prediction but also offer practical implications for improving asset pricing and portfolio management, with better Sharpe ratios and risk-adjusted returns demonstrated.

AI-Essentials

Dive into an insightful 1-hour video that explores how finance teams can significantly enhance their efficiency using AI and automation! This presentation, featuring interviews and discussions, is particularly helpful for those new to the practical applications of artificial intelligence in the financial sector. Whether you're just beginning to learn about AI or looking to understand its impact on financial operations, this video provides the perfect overview. 👇

Sponsor 👇

Investing in the next household name

RYSE has just launched in 100+ Best Buy stores, and you're in luck. You can still invest at only $1.50/share before their name becomes known nationwide. They’ve patented the only mass market shade automation device and their exclusive Best Buy deal resembles that which led to other past billion-dollar buyouts like Ring and Nest. Learn more + earn bonus shares →

Methodological Insights

“How to Incorporate Market Timing with ML-Predicted Volatility?”

Volatility timing is one of the best-known methods for timing investments. In this paper, they enhance this idea by simply improving volatility prediction with ML.

Here my takeaways: 👇

➡ The paper employs ML, notably model averaging and the LASSO method, to predict stock market volatility more accurately than traditional models. This enhanced forecasting capability is crucial for effective asset allocation.

➡ By integrating predictive information from various drivers, the research effectively forecasts market volatilities using high-dimensional models, which are shown to outperform standard volatility models in predicting performance.

➡ The paper highlights the construction of volatility timing portfolios that generally yield higher Sharpe ratios and risk-adjusted returns compared to static market portfolios, especially those timed with LASSO-based forecasts, across various forecasting horizons.

➡ It presents empirical results where volatility timing strategies surpass the static market portfolio in terms of average returns, with LASSO-driven portfolios achieving the best overall investment performance, indicating at least 24% to 42% higher Sharpe ratios than the market.

➡ Finally, the findings are summarized with visual evidence of cumulative log-returns (see image), showcasing that LASSO portfolios not only consistently outperform the market portfolio over a 14-year test period but also command the highest cumulative returns.

If you're enjoying our newsletter and want to support us, please recommend it to anyone you know who's interested in AI and Finance. Your referrals are the biggest compliment and help us grow! 🌟🤖💼