Hostname: page-component-5d84bcc8dc-jm5vp Total loading time: 0 Render date: 2026-09-11T11:49:42.626Z Has data issue: false hasContentIssue false
  • English
  • Français

Large and moderate deviations for Gaussian neural networks

Part of: Limit theorems

Published online by Cambridge University Press:  01 October 2025

Claudio Macci ,
Barbara Pacchiarotti  and
Giovanni Luca Torrisi
Claudio Macci*
Affiliation:
Università di Roma Tor Vergata
Barbara Pacchiarotti*
Affiliation:
Università di Roma Tor Vergata
Giovanni Luca Torrisi*
Affiliation:
Consiglio Nazionale delle Ricerche
*
*Postal address: Dipartimento di Matematica, Università di Roma Tor Vergata, Via della Ricerca Scientifica, I-00133 Rome, Italy.
*Postal address: Dipartimento di Matematica, Università di Roma Tor Vergata, Via della Ricerca Scientifica, I-00133 Rome, Italy.
****Postal address: Istituto per le Applicazioni del Calcolo (IAC), Consiglio Nazionale delle Ricerche, Via dei Taurini 19, I-00185 Rome, Italy. Email: giovanniluca.torrisi@cnr.it
Get access Rights & Permissions [Opens in a new window]

Abstract

We prove large and moderate deviations for the output of Gaussian fully connected neural networks. The main achievements concern deep neural networks (i.e. when the model has more than one hidden layer) and hold for bounded and continuous pre-activation functions. However, for deep neural networks fed by a single input, we have results even if the pre-activation is ReLU. When the network is shallow (i.e. there is exactly one hidden layer), the large and moderate principles hold for quite general pre-activation functions.

Keywords

Asymptotic behaviorcontraction principledeep neural networksReLU pre-activation function

MSC classification

Primary: 60F10: Large deviations 60F05: Central limit and other weak theorems

Information

Type
Original Article
Information
Journal of Applied Probability , Volume 63 , Issue 1 , March 2026 , pp. 96 - 115
Check for updates
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of Applied Probability Trust

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Apollonio, N., De Canditiis, D., Franzina, G., Stolfi, P. and Torrisi, G. L. (2024). Normal approximation of random Gaussian neural networks. Stoch. Syst. 15, 88–110.CrossRefGoogle Scholar
Balasubramanian, K., Goldstein, L., Ross, N. and Salim, A. (2024). Gaussian random field approximation via Stein’s method with applications to wide random neural networks. Appl. Comput. Harmonic Anal. 72, 101668.CrossRefGoogle Scholar
Basteri, A. and Trevisan, D. (2024). Quantitative Gaussian approximation of randomly initialized deep neural networks. Mach. Learn. 113, 1–31.CrossRefGoogle Scholar
Bathia, R. (1997). Matrix Analysis (Grad. Texts Math. 169). Springer, Berlin.Google Scholar
Bordino, A., Favaro, S. and Fortini, S. (2024). Non-asymptotic approximations of Gaussian neural networks via second-order Poincaré inequalities. Prog. Mach. Learn. Res. 253, 45–78.Google Scholar
Braun, A., Kohler, M., Langer, S. and Walk, H. (2024). Convergence rates for shallow neural networks learned by gradient descent. Bernoulli 30, 475–502.CrossRefGoogle Scholar
Cammarota, V., Marinucci, D., Salvi, M. and Vigogna, S. (2024). A quantitative functional central limit theorem for shallow neural networks. Mod. Stoch. Theory Appl. 11, 85–108.CrossRefGoogle Scholar
Chaganty, N. R. (1997). Large deviations for joint distributions and statistical applications. Sankhyā A 59, 147–166.Google Scholar
Dembo, A. and Zeitouni, O. (2010). Large Deviations Techniques and Applications (Stoch. Model. Appl. Prob. 38). Springer, Berlin.Google Scholar
Eldan, R., Mikulincer, D. and Schramm, T. (2021). Non-asymptotic approximations of neural networks by Gaussian processes. Proc. Mach. Learn. Res. 134, 1754–1775.Google Scholar
Favaro, S., Fortini, S. and Peluchetti, S. (2023). Deep stable neural networks: Large-width asymptotics and convergence rates. Bernoulli 29, 2574–2597.CrossRefGoogle Scholar
Favaro, S., Hanin, B., Marinucci, D., Nourdin, I. and Peccati, G. (2025). Quantitative CLTs in deep neural networks. Prob. Theory Relat. Fields 191, 933–977. https://doi.org/10.1007/s00440-025-01360-1.CrossRefGoogle Scholar
Giuliano, R., Macci, C. and Pacchiarotti, B. (2024). Asymptotic results for sums and extremes. J. Appl. Prob. 61, 1153–1171.CrossRefGoogle Scholar
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014). Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680.Google Scholar
Hanin, B. (2023). Random neural networks in the infinite width limit as Gaussian processes. Ann. Appl. Prob. 33, 4798–4819.CrossRefGoogle Scholar
Hanin, B. (2024). Random fully connected neural networks as perturbatively solvable hierarchies. J. Mach. Learn. Res. 25, 1–58.Google Scholar
Hirsch, C. and Willhalm, D. (2024). Large deviations of one-hidden-layer neural networks. Stoch. Dyn. 8, 2550002.CrossRefGoogle Scholar
Jung, P., Lee, H., Lee, J. and Yang, H. $\alpha$ -stable convergence of heavy-/light-tailed infinitely wide neural networks. Adv. Appl. Prob. 55, 1415–1441.CrossRefGoogle Scholar
LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning. Nature 521, 436–444.CrossRefGoogle ScholarPubMed
Li, B. and Saad, D. (2020). Large deviation analysis of function sensitivity in random deep neural networks. J. Phys. A 53, 104002.CrossRefGoogle Scholar
Neal, R. M. (1996). Priors for infinite networks. In Bayesian Learning for Neural Networks (Lect. Notes Statist. 118), ed. Neal, R. M.. Springer, New York, pp. 29–53.CrossRefGoogle Scholar
Roberts, D. A., Yaida, S. and Hanin, B. (2022). The Principles of Deep Learning Theory. Cambridge University Press.CrossRefGoogle Scholar
Saulis, L. and Statulevičius, V. A. (1991). Limit Theorems for Large Deviations (Math. Appl. (Sov. Ser.) 73). Kluwer, Dordrecht.Google Scholar
Sirignano, J. and Spiliopoulos, K. (2020). Mean field analysis of neural networks: A law of large numbers. SIAM J. Appl. Math. 80, 725–752.CrossRefGoogle Scholar
Vogel, Q. (2024). Large deviations of Gaussian neural networks with ReLU activation. Preprint, arXiv:2405.16958.Google Scholar
Zavatone-Veth, J. A. and Pehlevan, C. (2021). Exact marginal prior distributions of finite Bayesian neural networks. In Advances in Neural Information Processing Systems 34, eds. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang and J. Wortman Vaughan. Curran Associates, Red Hook, NY, pp. 3364–3375.Google Scholar