Hostname: page-component-848d4c4894-wzw2p Total loading time: 0 Render date: 2024-05-22T02:12:46.214Z Has data issue: false hasContentIssue false
  • English
  • Français

An ℓ1-oracle inequality for the Lasso in finite mixture Gaussian regression models

Published online by Cambridge University Press:  04 November 2013

Caroline Meynet
Caroline Meynet*
Affiliation:
Laboratoire de Mathématiques, Faculté des Sciences d’Orsay, Université Paris-Sud, 91405 Orsay, France. caroline.meynet@math.u-psud.fr
Get access

Abstract

We consider a finite mixture of Gaussian regression models for high-dimensional heterogeneous data where the number of covariates may be much larger than the sample size. We propose to estimate the unknown conditional mixture density by an ℓ1-penalized maximum likelihood estimator. We shall provide an ℓ1-oracle inequality satisfied by this Lasso estimator with the Kullback–Leibler loss. In particular, we give a condition on the regularization parameter of the Lasso to obtain such an oracle inequality. Our aim is twofold: to extend the ℓ1-oracle inequality established by Massart and Meynet [12] in the homogeneous Gaussian linear regression case, and to present a complementary result to Städler et al. [18], by studying the Lasso for its ℓ1-regularization properties rather than considering it as a variable selection procedure. Our oracle inequality shall be deduced from a finite mixture Gaussian regression model selection theorem for ℓ1-penalized maximum likelihood conditional density estimation, which is inspired from Vapnik’s method of structural risk minimization [23] and from the theory on model selection for maximum likelihood estimators developed by Massart in [11].

Keywords

Finite mixture of Gaussian regressions modelLassoℓ1-oracle inequalitiesmodel selection by penalizationℓ1-balls
Type
Research Article
Information
ESAIM: Probability and Statistics , Volume 17 , 2013 , pp. 650 - 671
Copyright
© EDP Sciences, SMAI, 2013

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

P.L. Bartlett, S. Mendelson and J. Neeman, ℓ 1-regularized linear regression: persistence and oracle inequalities, Probability and related fields. Springer (2011).
J.P. Baudry, Sélection de Modèle pour la Classification Non Supervisée. Choix du Nombre de Classes. Ph.D. thesis, Université Paris-Sud 11, France (2009).
Bickel, P.J., Ritov, Y. and Tsybakov, A.B., Simultaneous analysis of Lasso and Dantzig selector. Ann. Stat. 37 (2009) 1705–1732. Google Scholar
S. Boucheron, G. Lugosi and P. Massart, A non Asymptotic Theory of Independence. Oxford University press (2013).
Bühlmann, P. and van de Geer, S., On the conditions used to prove oracle results for the Lasso. Electr. J. Stat. 3 (2009) 1360–1392. Google Scholar
Candes, E. and Tao, T., The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35 (2007) 2313–2351. Google Scholar
S. Cohen and E. Le Pennec, Conditional Density Estimation by Penalized Likelihood Model Selection and Applications, RR-7596. INRIA (2011).
Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R., Least Angle Regression. Ann. Stat. 32 (2004) 407–499. Google Scholar
M. Hebiri, Quelques questions de sélection de variables autour de l’estimateur Lasso. Ph.D. Thesis, Université Paris Diderot, Paris 7, France (2009).
C. Huang, G.H.L. Cheang and A.R. Barron, Risk of penalized least squares, greedy selection and ℓ 1-penalization for flexible function librairies. Submitted to the Annals of Statistics (2008).
P. Massart, Concentration inequalities and model selection. Ecole d’été de Probabilités de Saint-Flour 2003. Lect. Notes Math. Springer, Berlin-Heidelberg (2007).
Massart, P. and Meynet, C., The Lasso as an ℓ 1-ball model selection procedure. Elect. J. Stat. 5 (2011) 669–687. Google Scholar
Maugis, C. and Michel, B., A non asymptotic penalized criterion for Gaussian mixture model selection. ESAIM: PS 15 (2011) 41–68. Google Scholar
G. McLachlan and D. Peel, Finite Mixture Models. Wiley, New York (2000).
Meinshausen, N. and Yu, B., Lasso type recovery of sparse representations for high dimensional data. Ann. Stat. 37 (2009) 246–270. Google Scholar
Redner, R.A. and Walker, H.F., Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev. 26 (1984) 195–239. Google Scholar
Rigollet, P. and Tsybakov, A., Exponential screening and optimal rates of sparse estimation. Ann. Stat. 39 (2011) 731–771. Google Scholar
Städler, N., Hlmann, B.P., and van de Geer, S., ℓ 1-penalization for mixture regression models. Test 19 (2010) 209–256.
Tibshirani, R., Regression shrinkage and selection via the Lasso. J. Roy. Stat. Soc. Ser. B 58 (1996) 267–288. Google Scholar
Osborne, M.R., Presnell, B. and Turlach, B.A., On the Lasso and its dual. J. Comput. Graph. Stat. 9 (2000) 319–337. Google Scholar
Osborne, M.R., Presnell, B. and Turlach, B.A, A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20 (2000) 389–404. Google Scholar
A. van der Vaart and J. Wellner, Weak Convergence and Empirical Processes. Springer, Berlin (1996).
V.N. Vapnik, Estimation of Dependencies Based on Empirical Data. Springer, New-York (1982).
V.N. Vapnik, Statistical Learning Theory. J. Wiley, New-York (1990).
Zhao, P. and Yu, B. On model selection consistency of Lasso. J. Mach. Learn. Res. 7 (2006) 2541–2563. Google Scholar