Skip to main content
    • Aa
    • Aa
  • Get access
    Check if you have access via personal or institutional login
  • Cited by 3
  • Cited by
    This article has been cited by the following publications. This list is generated based on data provided by CrossRef.

    Sahlin, U. Jeliazkova, N. and Öberg, T. 2014. Applicability Domain Dependent Predictive Uncertainty in QSAR Regressions. Molecular Informatics, Vol. 33, Issue. 1, p. 26.

    Toplak, Marko Močnik, Rok Polajnar, Matija Bosnić, Zoran Carlsson, Lars Hasselgren, Catrin Demšar, Janez Boyer, Scott Zupan, Blaž and Stålring, Jonna 2014. Assessment of Machine Learning Reliability Methods for Quantifying the Applicability Domain of QSAR Regression Models. Journal of Chemical Information and Modeling, Vol. 54, Issue. 2, p. 431.

    Bosnic, Zoran and Kononenko, Igor 2011. Integrations of Data Warehousing, Data Mining and Database Technologies.


Automatic selection of reliability estimates for individual regression predictions

  • Zoran Bosnić (a1) and Igor Kononenko (a1)
  • DOI:
  • Published online: 01 March 2010

In machine learning and its risk-sensitive applications (e.g. medicine, engineering, business), the reliability estimates for individual predictions provide more information about the individual prediction error (the difference between the true label and regression prediction) than the average accuracy of predictive model (e.g. relative mean squared error). Furthermore, they enable the users to distinguish between more and less reliable predictions. The empirical evaluations of the existing individual reliability estimates revealed that the successful estimates’ performance depends on the used regression model and on the particular problem domain. In the current paper, we focus on that problem as such and propose and empirically evaluate two approaches for automatic selection of the most appropriate estimate for a given domain and regression model: the internal cross-validation approach and the meta-learning approach. The testing results of both approaches demonstrated an advantage in the performance of dynamically chosen reliability estimates to the performance of the individual reliability estimates. The best results were achieved using the internal cross-validation procedure, where reliability estimates significantly positively correlated with the prediction error in 73% of experiments. In addition, the preliminary testing of the proposed methodology on a medical domain demonstrated the potential for its usage in practice.

Corresponding author
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Z. Bosnić , I. Kononenko 2008a. Estimation of regressor reliability. Journal of Intelligent Systems 17(1/3), 297311.

L. Breiman 1996. Bagging predictors. Machine Learning 24(2), 123140.

L. Breiman 2001. Random forests. Machine Learning 45(1), 532.

R. Caruana 1997. Multitask learning. Machine Learning 28(1), 4175.

M. J. Crowder , A. C. Kimber , R. L. Smith , T. J. Sweeting 1991. Statistical Concepts in Reliability. Statistical Analysis of Reliability Data. Chapman & Hall.

Y. Freund , R. E. Schapire 1997. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119139.

G. Giacinto , F. Roli 2001. Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34(9), 18791881.

B. Jeon , D. A. Landgrebe 1994. Parzen density estimation using clustering-based branch and bound. IEEE Transactions on Pattern Analysis and Machine Intelligence, 950954.

I. Kononenko , M. Kukar 2007. Machine Learning and Data Mining: Introduction to Principles and Algorithms. Horwood Publishing Limited.

A. M. Krieger , P. E. Green 1999. A cautionary note on using internal cross validation to select the number of clusters. Psychometrika 64, 341353.

M. Li , P. Vitányi 1993. An Introduction to Kolmogorov Complexity and its Applications. Springer-Verlag.

S. Schaal , C. G. Atkeson 1998. Constructive incremental learning from only local information. Neural Computation 10(8), 20472084.

B. W. Silverman 1986. Density Estimation for Statistics and Data Analysis. Monographs on Statistics and Applied Probability. Chapman and Hall.

R. Tibshirani , K. Knight 1999. Model search and inference by bootstrap bumping. Journal of Computational and Graphical Statistics 8, 671686.

V. Vapnik 1995. The Nature of Statistical Learning Theory. Springer.

R. Vilalta , Y. Drissi 2002. A perspective view and survey of metalearning. Artificial Intelligence Review 18(2), 7795.

M. P. Wand , M. C. Jones 1995. Kernel Smoothing. Chapman and Hall.

D. H. Wolpert 1992. Stacked generalization. In Neural Networks, Amari S. Grossberg S. & Taylor J. G. (eds) 5, 241259. Pergamon Press.

K. Woods , W. P. Kegelmeyer , K. Bowyer 1997. Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on PAMI 19(4), 405410.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Knowledge Engineering Review
  • ISSN: 0269-8889
  • EISSN: 1469-8005
  • URL: /core/journals/knowledge-engineering-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *