Skip to main content
    • Aa
    • Aa

Automatic selection of reliability estimates for individual regression predictions

  • Zoran Bosnić (a1) and Igor Kononenko (a1)

In machine learning and its risk-sensitive applications (e.g. medicine, engineering, business), the reliability estimates for individual predictions provide more information about the individual prediction error (the difference between the true label and regression prediction) than the average accuracy of predictive model (e.g. relative mean squared error). Furthermore, they enable the users to distinguish between more and less reliable predictions. The empirical evaluations of the existing individual reliability estimates revealed that the successful estimates’ performance depends on the used regression model and on the particular problem domain. In the current paper, we focus on that problem as such and propose and empirically evaluate two approaches for automatic selection of the most appropriate estimate for a given domain and regression model: the internal cross-validation approach and the meta-learning approach. The testing results of both approaches demonstrated an advantage in the performance of dynamically chosen reliability estimates to the performance of the individual reliability estimates. The best results were achieved using the internal cross-validation procedure, where reliability estimates significantly positively correlated with the prediction error in 73% of experiments. In addition, the preliminary testing of the proposed methodology on a medical domain demonstrated the potential for its usage in practice.

Corresponding author
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Z. Bosnić , I. Kononenko 2008a. Estimation of regressor reliability. Journal of Intelligent Systems 17(1/3), 297311.

L. Breiman 1996. Bagging predictors. Machine Learning 24(2), 123140.

R. Caruana 1997. Multitask learning. Machine Learning 28(1), 4175.

M. J. Crowder , A. C. Kimber , R. L. Smith , T. J. Sweeting 1991. Statistical Concepts in Reliability. Statistical Analysis of Reliability Data. Chapman & Hall.

Y. Freund , R. E. Schapire 1997. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119139.

G. Giacinto , F. Roli 2001. Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34(9), 18791881.

B. Jeon , D. A. Landgrebe 1994. Parzen density estimation using clustering-based branch and bound. IEEE Transactions on Pattern Analysis and Machine Intelligence, 950954.

I. Kononenko , M. Kukar 2007. Machine Learning and Data Mining: Introduction to Principles and Algorithms. Horwood Publishing Limited.

A. M. Krieger , P. E. Green 1999. A cautionary note on using internal cross validation to select the number of clusters. Psychometrika 64, 341353.

M. Li , P. Vitányi 1993. An Introduction to Kolmogorov Complexity and its Applications. Springer-Verlag.

S. Schaal , C. G. Atkeson 1998. Constructive incremental learning from only local information. Neural Computation 10(8), 20472084.

B. W. Silverman 1986. Density Estimation for Statistics and Data Analysis. Monographs on Statistics and Applied Probability. Chapman and Hall.

R. Tibshirani , K. Knight 1999. Model search and inference by bootstrap bumping. Journal of Computational and Graphical Statistics 8, 671686.

V. Vapnik 1995. The Nature of Statistical Learning Theory. Springer.

R. Vilalta , Y. Drissi 2002. A perspective view and survey of metalearning. Artificial Intelligence Review 18(2), 7795.

M. P. Wand , M. C. Jones 1995. Kernel Smoothing. Chapman and Hall.

D. H. Wolpert 1992. Stacked generalization. In Neural Networks, Amari S. Grossberg S. & Taylor J. G. (eds) 5, 241259. Pergamon Press.

K. Woods , W. P. Kegelmeyer , K. Bowyer 1997. Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on PAMI 19(4), 405410.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Knowledge Engineering Review
  • ISSN: 0269-8889
  • EISSN: 1469-8005
  • URL: /core/journals/knowledge-engineering-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *