Likert items and questionnaires are widely used in HCI, particularly to measure user experience. However, there is some confusion over which is the right test to use to analyse data arising from these instruments. Furthermore, this book has proposed several more modern alternatives to traditional statistical tests but there is little evidence if they are better in the context of this particular sort of data. This chapter therefore reports on several simulation studies to compare the variety of tests that can be used to analyse Likert item and questionnaire data. The results suggest that this sort of data best reveals dominance effects and therefore that tests of dominance are the most suitable, and most robust, tests to use.

Bayesian statistics are often presented as a better, modern alternative to the Frequentist approaches centred around NHST and the resulting obsession with statistical significance. This chapter outlines the basic ideas of Frequentist and Bayesian statistics. It raises critiques of the Frequentist approach but also points out constraints on the Bayesian approach that are often omitted or overlooked. In particular, the chapter discusses how both Bayesian and Frequentist approaches rely on a move from statistical hypotheses to substantive hypotheses that cannot be justified by consideration of the statistics alone. Instead, both approaches can lead to sound knowledge through a care for data analysis, tied to the experiments that generate the data.

Many statistical tests that are commonly used rely on the assumption that data are normally distributed. This chapter discusses why normality commonly occurs in statistics but also how, in many practical situations in HCI, it is not safe to assume normality. It also shows how tests for normality are not meaningful or useful. Instead, where normality is in doubt, analysis should be more careful and use suitable alternative tests.

Effects and the size of effects are proposed as a key way to overcome the limitations of significance testing in modern statistics. However, traditionally, effects are usually added on to the statistical testing procedure. This chapter proposes instead thinking first about what effects to look for in the context of the research and its maturity within the discipline of HCI. The main types of effect, changes in location, stochastic dominance and (co)variation, are described. Consideration of which of these effects that current best knowledge can predict leads to choosing the test most suitable to find those effects.

Cronbach's alpha is often used as a key measure of reliability in questionnaires developed in HCI. However, the state-of-the-art literature on Cronbach's alpha shows that it is itself not a reliable measure. This chapter demonstrates the problems and leads to a more nuanced interpretation of reliability, using both Cronbach's alpha and other more modern alternatives.

The focus of statistical testing on significance can lead to the two types of error discussed in this chapter: finding a significant result when there is no underlying real effect; failing to get significance when there is in fact an underlying real effect. Though a blinkered approach to significance can be problematic for good data analysis, consideration of the two types of error is useful first to help think about sample sizes in studies but, second, it is a good way to assess the quality and robustness of statistical tests.

Factor analysis is quite regularly seen in HCI to validate questionnaires, particularly those used to measure user experience. This chapter describes and illustrates the process of developing a questionnaire to measure a particular concept, starting from item generation through to factor analysis and interpretation. It introduces bifactor analysis as a more recent development in factor analysis that could help the interpretation of questionnaires made up of multiple factors.

A directional hypothesis predicts the specific way in which data are affected by an experimental manipulation. This, therefore, fits well with severe testing by making a concrete, explicit prediction. However, one-tailed testing associated with directional hypotheses regularly receives criticism because it can look like a way to make it easy to achieve statistical significance. This chapter describes why this is a problem and makes the case that despite the fit with severe testing, two-tailed testing is still the better way to do statistical analysis.