Hostname: page-component-6766d58669-bp2c4 Total loading time: 0 Render date: 2026-05-15T12:36:56.302Z Has data issue: false hasContentIssue false
  • English
  • Français

Estimating Stock Market Betas via Machine Learning

Published online by Cambridge University Press:  08 February 2024

Wolfgang Drobetz ,
Fabian Hollstein Open the ORCID record for Fabian Hollstein [Opens in a new window] ,
Tizian Otto  and
Marcel Prokopczuk
Wolfgang Drobetz*
Affiliation:
University of Hamburg, Faculty of Business Administration
Fabian Hollstein
Affiliation:
Saarland University, School of Human and Business Sciences fabian.hollstein@uni-saarland.de
Tizian Otto
Affiliation:
University of Hamburg, Faculty of Business Administration tizian.otto@uni-hamburg.de
Marcel Prokopczuk
Affiliation:
Leibniz University, Hannover School of Economics and Management prokopczuk@fcm.uni-hannover.de
*
wolfgang.drobetz@uni-hamburg.de (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.

Machine learning-based stock market beta estimators outperform established benchmark models both statistically and economically. Analyzing the predictability of time-varying market betas of U.S. stocks, we document that machine learning-based estimators produce the lowest forecast and hedging errors. They also help to create better market-neutral anomaly strategies and minimum variance portfolios. Among the various techniques, random forests perform the best overall. Model complexity is highly time-varying. Historical stock market betas, turnover, and size are the most important predictors. Compared to linear regressions, allowing for nonlinearity and interactions significantly improves predictive performance.

Information

Type
Research Article
Information
Journal of Financial and Quantitative Analysis , Volume 60 , Issue 3 , May 2025 , pp. 1074 - 1110
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of the Michael G. Foster School of Business, University of Washington

Footnotes

We thank two anonymous referees, Yakov Amihud, Michael Bauer, Wolfgang Bessler, Axel Cabrol, Mikhail Chernov, Hubert Dichtl, Thierry Foucault (the editor), Kay Giesecke, Lisa Goldberg, Valentin Haddad, Barney Hartman-Glaser, Bernard Herskovic, Bryan Kelly, Serhiy Kozak, Markus Leippold, Martin Lettau, Lars A. Lochstoer, Harald Lohre, Tyler Muir, Andreas Neuhierl, Terrance Odean, Stavros Panageas, Markus Pelger, Tatjana Puhan, Carsten Rother, Boris Vallee, Michael Weber, and Ivo Welch for helpful comments and suggestions.

References

Amihud, Y., and Mendelson, H.. “The Liquidity Route to a Lower Cost of Capital.” Journal of Applied Corporate Finance, 12 (2000), 8–25.CrossRefGoogle Scholar
Andersen, T. G.; Bollerslev, T.; Diebold, F. X.; and Wu, G.. “Realized Beta: Persistence and Predictability.” In Advances in Econometrics: Econometric Analysis of Economic and Financial Time Series, Fomby, T., and Terrel, D., eds. Amsterdam, The Netherlands: Elsevier (2006), 1–39.Google Scholar
Ang, A., and Chen, J.. “CAPM Over the Long Run: 1926–2001.” Journal of Empirical Finance, 14 (2007), 1–40.CrossRefGoogle Scholar
Bailey, D. H.; Borwein, J. M.; de Prado, M. L.; and Zhu, Q. J.. “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.” Notices of the American Mathematical Society, 61 (2014), 458–471.CrossRefGoogle Scholar
Bailey, D. H.; Borwein, J. M.; de Prado, M. L.; and Zhu, Q. J.. “The Probability of Backtest Overfitting.” Journal of Computational Finance, 20 (2017), 1460–1559.Google Scholar
Bali, T. G.; Goyal, A.; Huang, D.; Jiang, F.; and Wen, Q.. “The Cross-Sectional Pricing of Corporate Bonds Using Big Data and Machine Learning.” Working Paper, Georgetown University (2022).Google Scholar
Barber, B. M.; Huang, X.; and Odean, T.. “Which Factors Matter to Investors? Evidence from Mutual Fund Flows.” Review of Financial Studies, 29 (2016), 2600–2642.CrossRefGoogle Scholar
Beaver, W.; Kettler, P.; and Scholes, M.. “The Association Between Market Determined and Accounting Determined Risk Measures.” Accounting Review, 45 (1970), 654–682.Google Scholar
Becker, J.; Hollstein, F.; Prokopczuk, M.; and Sibbertsen, P.. “The Memory of Beta.” Journal of Banking and Finance, 124 (2021), 106026.CrossRefGoogle Scholar
Berk, J. B., and van Binsbergen, J. H.. “Assessing Asset Pricing Models Using Revealed Preference.” Journal of Financial Economics, 119 (2016), 1–23.CrossRefGoogle Scholar
Bianchi, D.; Büchner, M.; and Tamoni, A.. “Bond Risk Premiums with Machine Learning.” Review of Financial Studies, 34 (2021), 1046–1089.CrossRefGoogle Scholar
Black, F.; Jensen, M. C.; and Scholes, M. S.. “The Capital Asset Pricing Model: Some Tests.” In Studies in the Theory of Capital Markets, Jensen, M. C., ed. New York, NY: Praeger (1972), 79–121.Google Scholar
Bollerslev, T.; Engle, R. F.; and Wooldridge, J. M.. “A Capital Asset Pricing Model with Time-Varying Covariances.” Journal of Political Economy, 96 (1988), 116–131.CrossRefGoogle Scholar
Bucci, A. “Realized Volatility Forecasting with Neural Networks.” Journal of Financial Econometrics, 18 (2020), 502–531.CrossRefGoogle Scholar
Campbell, J. Y.; Lettau, M.; Malkiel, B. G.; and Xu, Y.. “Have Individual Stocks Become More Volatile? An Empirical Exploration of Idiosyncratic Risk.” Journal of Finance, 56 (2001), 1–43.CrossRefGoogle Scholar
Chan, L. K.; Karceski, J.; and Lakonishok, J.. “On Portfolio Optimization: Forecasting Covariances and Choosing the Risk Model.” Review of Financial Studies, 12 (1999), 937–974.CrossRefGoogle Scholar
Chincarini, L. B.; Kim, D.; and Moneta, F.. “Beta and Firm Age.” Journal of Empirical Finance, 58 (2020), 50–74.CrossRefGoogle Scholar
Christensen, K.; Siggaard, M.; and Veliyev, B.. “A Machine Learning Approach to Volatility Forecasting.” Journal of Financial Econometrics, 21 (2023), 1680–1727.CrossRefGoogle Scholar
Cochrane, J. H. “Presidential Address: Discount Rates.” Journal of Finance, 66 (2011), 1047–1108.CrossRefGoogle Scholar
Connor, G.; Hagmann, M.; and Linton, O.. “Efficient Semiparametric Estimation of the Fama–French Model and Extensions.” Econometrica, 80 (2012), 713–754.Google Scholar
Connor, G., and Linton, O.. “Semiparametric Estimation of a Characteristic-Based Factor Model of Common Stock Returns.” Journal of Empirical Finance, 14 (2007), 694–717.CrossRefGoogle Scholar
Cosemans, M.; Frehen, R.; Schotman, P. C.; and Bauer, R.. “Estimating Market Betas Using Prior Information Based on Firm Fundamentals.” Review of Financial Studies, 29 (2016), 1072–1112.CrossRefGoogle Scholar
Daniel, K., and Moskowitz, T. J.. “Momentum Crashes.” Journal of Financial Economics, 122 (2016), 221–247.CrossRefGoogle Scholar
Daniel, K.; Mota, L.; Rottke, S.; and Santos, T.. “The Cross-Section of Risk and Returns.” Review of Financial Studies, 33 (2020), 1927–1979.CrossRefGoogle Scholar
Diebold, F., and Mariano, R.. “Comparing Predictive Accuracy.” Journal of Business and Economic Statistics, 13 (1995), 253–263.CrossRefGoogle Scholar
Donaldson, R. G., and Kamstra, M.. “An Artificial Neural Network-GARCH Model for International Stock Return Volatility.” Journal of Empirical Finance, 4 (1997), 17–46.CrossRefGoogle Scholar
Drobetz, W., and Otto, T.. “Empirical Asset Pricing via Machine Learning: Evidence from the European Stock Market.” Journal of Asset Management, 22 (2021), 507–538.CrossRefGoogle Scholar
Fama, E. F., and French, K. R.. “The Cross-Section of Expected Stock Returns.” Journal of Finance, 47 (1992), 427–465.Google Scholar
Fama, E. F., and French, K. R.. “Common Risk Factors in the Returns on Stocks and Bonds.” Journal of Financial Economics, 33 (1993), 3–56.CrossRefGoogle Scholar
Fama, E. F., and French, K. R.. “Industry Costs of Equity.” Journal of Financial Economics, 43 (1997), 153–193.CrossRefGoogle Scholar
Fama, E. F., and French, K. R.. “Dissecting Anomalies.” Journal of Finance, 63 (2008), 1653–1678.CrossRefGoogle Scholar
Fama, E. F., and French, K. R.. “A Five-Factor Asset Pricing Model.” Journal of Financial Economics, 116 (2015), 1–22.CrossRefGoogle Scholar
Fama, E. F., and MacBeth, J. D.. “Risk, Return, and Equilibrium: Tests.” Journal of Political Economy, 81 (1973), 607–636.CrossRefGoogle Scholar
Fan, J.; Liao, Y.; and Wang, W.. “Projected Principal Component Analysis in Factor Models.” Annals of Statistics, 44 (2016), 219–254.CrossRefGoogle ScholarPubMed
Fernandes, M.; Medeiros, M. C.; and Scharth, M.. “Modeling and Predicting the CBOE Market Volatility Index.” Journal of Banking and Finance, 40 (2014), 1–10.CrossRefGoogle Scholar
Ferson, W. E., and Harvey, C. R.. “Conditioning Variables and the Cross Section of Stock Returns.” Journal of Finance, 54 (1999), 1325–1360.CrossRefGoogle Scholar
Frazzini, A., and Pedersen, L. H.. “Betting Against Beta.” Journal of Financial Economics, 111 (2014), 1–25.CrossRefGoogle Scholar
Freyberger, J.; Höppner, B.; Neuhierl, A.; and Weber, M.. “Missing Data in Asset Pricing Panels.” Review of Financial Studies, forthcoming (2024).CrossRefGoogle Scholar
Freyberger, J.; Neuhierl, A.; and Weber, M.. “Dissecting Characteristics Nonparametrically.” Review of Financial Studies, 33 (2020), 2326–2377.CrossRefGoogle Scholar
Ghysels, E., and Jacquier, E.. “Market Beta Dynamics and Portfolio Efficiency.” Working Paper, Boston University (2006).CrossRefGoogle Scholar
Giacomini, R., and White, H.. “Tests of Conditional Predictive Ability.” Econometrica, 74 (2006), 1545–1578.CrossRefGoogle Scholar
Gilbert, T.; Hrdlicka, C.; Kalodimos, J.; and Siegel, S.. “Daily Data is Bad for Beta: Opacity and Frequency-Dependent Betas.” Review of Asset Pricing Studies, 4 (2014), 78–117.CrossRefGoogle Scholar
Graham, J. R. “Presidential Address: Corporate Finance and Reality.” Journal of Finance, 77 (2022), 1975–2049.CrossRefGoogle Scholar
Graham, J. R., and Harvey, C. R.. “The Theory and Practice of Corporate Finance: Evidence from the Field.” Journal of Financial Economics, 60 (2001), 187–243.CrossRefGoogle Scholar
Grundy, B. D., and Martin, J. S. M.. “Understanding the Nature of the Risks and the Source of the Rewards to Momentum Investing.” Review of Financial Studies, 14 (2001), 29–78.CrossRefGoogle Scholar
Gu, S.; Kelly, B.; and Xiu, D.. “Empirical Asset Pricing via Machine Learning.” Review of Financial Studies, 33 (2020), 2223–2273.CrossRefGoogle Scholar
Gu, S.; Kelly, B.; and Xiu, D.. “Autoencoder Asset Pricing Models.” Journal of Econometrics, 222 (2021), 429–450.CrossRefGoogle Scholar
Gulen, H.; Xing, Y.; and Zhang, L.. “Value Versus Growth: Time-Varying Expected Stock Returns.” Financial Management, 40 (2011), 381–407.CrossRefGoogle Scholar
Hansen, P. R., and Lunde, A.. “Consistent Ranking of Volatility Models.” Journal of Econometrics, 131 (2006), 97–121.CrossRefGoogle Scholar
Hansen, P. R.; Lunde, A.; and Nason, J. M.. “The Model Confidence Set.” Econometrica, 79 (2011), 453–497.Google Scholar
Harvey, C. R., and Liu, Y.. “Evaluating Trading Strategies.” Journal of Portfolio Management, 40 (2014), 108–118.Google Scholar
Harvey, C. R., and Liu, Y.. “Backtesting.” Journal of Portfolio Management, 42 (2015), 13–28.Google Scholar
Harvey, C. R.; Liu, Y.; and Zhu, H.. “… and the Cross-Section of Expected Returns.” Review of Financial Studies, 29 (2016), 5–68.CrossRefGoogle Scholar
Hillebrand, E., and Medeiros, M. C.. “The Benefits of Bagging for Forecast Models of Realized Volatility.” Econometric Reviews, 29 (2010), 571–593.CrossRefGoogle Scholar
Hollstein, F., and Prokopczuk, M.. “Estimating Beta.” Journal of Financial and Quantitative Analysis, 51 (2016), 1437–1466.CrossRefGoogle Scholar
Hollstein, F.; Prokopczuk, M.; and Simen, C. Wese. “Estimating Beta: Forecast Adjustments and the Impact of Stock Characteristics for a Broad Cross-Section.” Journal of Financial Markets, 44 (2019), 91–118.CrossRefGoogle Scholar
Hou, K.; Xue, C.; and Zhang, L.. “Replicating Anomalies.” Review of Financial Studies, 33 (2020), 2019–2133.CrossRefGoogle Scholar
Jacobs, M. T., and Shivdasani, A.. “Do You Know Your Cost of Capital?” Harvard Business Review, 90 (2012), 118–124.Google Scholar
Jacoby, G.; Fowler, D. J.; and Gottesman, A. A.. “The Capital Asset Pricing Model and the Liquidity Effect: A Theoretical Approach.” Journal of Financial Markets, 3 (2000), 69–81.CrossRefGoogle Scholar
Jagannathan, R., and Wang, Z.. “The Conditional CAPM and the Cross-Section of Expected Returns.” Journal of Finance, 51 (1996), 3–53.Google Scholar
Jourovski, A.; Dubikovskyy, V.; Adell, P.; Ramakrishnan, R.; and Kosowski, R.. “Forecasting Beta Using Machine Learning and Equity Sentiment Variables.” In Machine Learning for Asset Management: New Developments and Financial Applications, Jurczenko, E., ed. London, UK: Wiley-ISTE (2020), 231–260.CrossRefGoogle Scholar
Karolyi, G. A. “Predicting Risk: Some New Generalizations.” Management Science, 38 (1992), 57–74.CrossRefGoogle Scholar
Kelly, B.; Moskowitz, T. J.; and Pruitt, S.. “Understanding Momentum and Reversal.” Journal of Financial Economics, 140 (2021), 726–743.CrossRefGoogle Scholar
Kelly, B. T.; Pruitt, S.; and Su, Y.. “Characteristics are Covariances: A Unified Model of Risk and Return.” Journal of Financial Economics, 134 (2019), 501–524.CrossRefGoogle Scholar
Kim, S.; Korajczyk, R. A.; and Neuhierl, A.. “Arbitrage Portfolios.” Review of Financial Studies, 34 (2020), 2813–2856.CrossRefGoogle Scholar
Kozak, S.; Nagel, S.; and Santosh, S.. “Shrinking the Cross-Section.” Journal of Financial Economics, 135 (2020), 271–292.CrossRefGoogle Scholar
Leippold, M.; Wang, Q.; and Zhou, W.. “Machine Learning in the Chinese Stock Market.” Journal of Financial Economics, 145 (2021), 64–82.CrossRefGoogle Scholar
Lewellen, J. “The Cross-Section of Expected Stock Returns.” Critical Finance Review, 4 (2015), 1–44.CrossRefGoogle Scholar
Lintner, J. “Security Prices, Risk, and Maximal Gains from Diversification.” Journal of Finance, 20 (1965), 587–615.Google Scholar
Lo, A. W., and MacKinlay, A. C.. “Data-Snooping Biases in Tests of Financial Asset Pricing Models.” Review of Financial Studies, 3 (1990), 431–467.CrossRefGoogle Scholar
Luong, C., and Dokuchaev, N.. “Forecasting of Realised Volatility with the Random Forests Algorithm.” Journal of Risk and Financial Management, 11 (2018), 61.CrossRefGoogle Scholar
Mincer, J. A., and Zarnowitz, V.. “The Evaluation of Economic Forecasts.” In Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, Mincer, J. A., ed. Cambridge, MA: NBER (1969), 3–46.Google Scholar
Mittnik, S.; Robinzonov, N.; and Spindler, M.. “Stock Market Volatility: Identifying Major Drivers and the Nature of Their Impact.” Journal of Banking and Finance, 58 (2015), 1–14.CrossRefGoogle Scholar
Mossin, J. “Equilibrium in a Capital Asset Market.” Econometrica, 34 (1966), 768–783.CrossRefGoogle Scholar
Newey, W. K., and West, K. D.. “A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix.” Econometrica, 55 (1987), 703–708.CrossRefGoogle Scholar
Novy-Marx, R. “Operating Leverage.” Review of Finance, 15 (2011), 103–134.CrossRefGoogle Scholar
Novy-Marx, R., and Velikov, M.. “Betting Against Betting Against Beta.” Journal of Financial Economics, 143 (2022), 80–106.CrossRefGoogle Scholar
Patton, A. J. “Volatility Forecast Comparison Using Imperfect Volatility Proxies.” Journal of Econometrics, 160 (2011), 246–256.CrossRefGoogle Scholar
Petkova, R., and Zhang, L.. “Is Value Riskier Than Growth?” Journal of Financial Economics, 78 (2005), 187–202.CrossRefGoogle Scholar
Rahimikia, E., and Poon, S. H.. “Machine Learning for Realised Volatility Forecasting.” Working Paper, University of Manchester (2020).CrossRefGoogle Scholar
Schorfheide, F., and Wolpin, K. I.. “On the Use of Holdout Samples for Model Selection.” American Economic Review, 102 (2012), 477–481.CrossRefGoogle Scholar
Sharpe, W. F. “Capital Asset Prices: A Theory of Market Equilibrium Under Conditions of Risk.” Journal of Finance, 19 (1964), 425–442.Google Scholar
Vasicek, O. “A Note on Using Cross-Sectional Information in Bayesian Estimation of Market Betas.” Journal of Finance, 28 (1973), 1233–1239.Google Scholar
Van Binsbergen, J. H.; Han, X.; and Lopez-Lira, A.. “Man vs. Machine Learning: The Term Structure of Earnings Expectations and Conditional Biases.” Review of Financial Studies, 36 (2023), 2361–2396.CrossRefGoogle Scholar
Welch, I. “Simply Better Market Betas.” Critical Finance Review, 11 (2022), 37–64.CrossRefGoogle Scholar
West, K. D. “Forecast Evaluation.” In Handbook of Economic Forecasting, Elliott, G., Granger, C., and Timmermann, A., eds. Amsterdam, The Netherlands: Elsevier (2006), 99–134.CrossRefGoogle Scholar
Supplementary material: File

Drobetz et al. supplementary material

Drobetz et al. supplementary material
Download Drobetz et al. supplementary material(File)
File 1.3 MB