The first several chapters of this book were introductory in style, though with more attention to the subtleties of practical application than is common in elementary treatments. The simple models considered here are somewhat restricted in their range of application, because they are based on assumptions of independence and stationarity which will not nearly always be true. The last several chapters treated topics that have traditionally been thought of as more advanced. We hope this has given readers a taste of models for data which can be applied in a wide variety of situations.

Models that are not strictly correct, or even perhaps badly broken, may nevertheless be useful, giving acceptably accurate predictions. The validity of model assumptions remains an important issue. We need to know the limits of the usefulness of our models. Comparing results from a simple model with results from a model that takes better account of the data can help in developing intuition. Somewhat ironically, Chapters 7-13 could be viewed as essential background for those who hope to do a good job of the analyses described in Chapters 3-6! An understanding of multi-level and repeated measures models seems particularly important, since these models bring attention to structure in the noise component which must be accounted for in a proper analysis.

In straight line regression, a response variable y is regressed on a single explanatory variable x. Multiple linear regression generalizes this methodology to allow multiple explanatory or predictor variables. The focus may be on accurate prediction. Or it may, alternatively or additionally, be on the regression coefficients themselves. Be warned that interpreting the regression coefficients is not as straightforward as it might appear.

The discussion will emphasize the use of model diagnostics, and of graphs that give insight into the model fit. Diagnostic checks are intended to assist in the tuning of models so that they perform well when used for their intended purpose, and do not give unexpected and perhaps unpleasant surprises. For example, a model fit that is unduly affected by influential outliers may give results that are less than satisfactory with the main body of the data. This is one of several common types of departure from model assumptions that diagnostic checks may bring to attention.

Basic ideas: a book weight example

The book weight example has two x-variables in the regression equation. In the data shown in Figure 6.1 and printed to the right of the figure, seven books with hardback covers have been added to the eight softbacks. Code for the figure is:

## Plot weight vs volume: data frame allbacks (DAAG)

plot(weight ˜ volume, data=allbacks, pch=c(16,1)[unclass(cover)])

# unclass(cover) gives the integer codes that identify levels with(allbacks, text(weight ˜ volume, labels=paste(1:15), pos=c(2,4)[unclass(cover)]))

Explanatory variables are the volume of the book ignoring the covers, and the total area of the front and back covers.

Earlier chapters have made extensive use of exploratory graphs that have examined variables and their pair wise relationships, as a preliminary to regression modeling. Scatter plot matrices have been an important tool, and will be used in this chapter also. The focus will move from regression to methods that seek insight into the pattern presented by multiple variables. While the methodology has applications in a regression context, this is not a primary focus.

There are a number of methods that project the data on to a low-dimensional space, commonly two dimensions, suggesting “views” of the data that may be insightful. In the absence of other sources of guidance, it is reasonable to begin with views that have been thus suggested. One of the most widely used methods for projecting onto a low-dimensional space is principal components analysis (PCA).

The PCA form of mathematical representation has applications in many contexts beyond those discussed here. As used here, PCA is a special case of a much wider class of multidimensional scaling (MDS) methods. Subsection 12.1.3 is a brief introduction to this wider class of methods.

Principal components analysis replaces the input variables by new derived variables, called principal components. The analysis orders the principal components according to the amounts that they contribute to the total of the variances of the original variables.

The model matrix X is fundamental to all calculations for a linear model. The model matrix carries the information needed to calculate the fitted values that correspond to any particular choice of coefficients. There is a one-to-one correspondence between columns of X and regression coefficients.

In Chapter 6, the columns of the model matrix contained the observed values of the explanatory variables, perhaps after transformation. Fitted values were obtained by multiplying the first column by the first coefficient (usually the intercept), the second column by the second coefficient, and so on across all columns. The sum of the products in any row is the fitted value for that row.

This chapter will explore new ways to relate the columns of the model matrix to the explanatory variables, where a variable may be either a vector of numeric values, or a factor. Vectors of zeros and ones (columns of “dummy” variables) can be used to handle factor levels, but as noted below there are other possibilities. For modeling a quadratic form of response, we take values of x as one of the columns and values of x2 as another. The model matrix framework also allows the modeling of many other forms of non-linear response. As before, the regression calculations find the set of coefficients that best predicts the observed responses, in the sense of minimizing the sum of squares of the residuals.

What is the best way to begin investigation of a new set of data? What forms of data exploration will draw attention to obvious errors or quirks in the data, or to obvious clues that the data contain? What checks are desirable before proceeding with an intended formal analysis, or to help decide what formal analysis may be appropriate? What can be learned from investigations that other researchers have done with similar data?

Competent statisticians have always used graphs to check their data. Numerical summaries, such as an average, can be very useful, but important features of the data may be missed without a glance at an appropriate graph. Careful consideration may be needed to choose a graph that will be effective for the purpose in hand.

We will see in Chapter 3 that an integral part of statistical analysis is the development of a model that accurately describes the data, clarifies what the data say, and makes prediction possible. Without model assumptions, there cannot be a meaningful formal analysis! As assumptions are strengthened, the chances of getting clear results improve. The price for stronger assumptions is that, if wrong, the results may be wrong. Graphical techniques have been developed for checking, to the extent possible, many of the assumptions that must be made in practice.

This book is an exposition of statistical methodology that focuses on ideas and concepts, and makes extensive use of graphical presentation. It avoids, as much as possible, the use of mathematical symbolism. It is particularly aimed at scientists who wish to do statistical analyses on their own data, preferably with reference as necessary to professional statistical advice. It is intended to complement more mathematically oriented accounts of statistical methodology. It may be used to give students with a more specialist statistical interest exposure to practical data analysis.

While no prior knowledge of specific statistical methods or theory is assumed, there is a demand that readers bring with them, or quickly acquire, some modest level of statistical sophistication. Readers should have some prior exposure to statistical methodology, some prior experience of working with real data, and be comfortable with the typing of analysis commands into the computer console. Some prior familiarity with regression and with analysis of variance will be helpful.

We cover a range of topics that are important for many different areas of statistical application. As is inevitable in a book that has this broad focus, there will be investigators working in specific areas – perhaps epidemiology, or psychology, or sociology, or ecology – who will regret the omission of some methodologies that they find important.

We comment extensively on analysis results, noting inferences that seem well-founded, and noting limitations on inferences that can be drawn. We emphasize the use of graphs for gaining insight into data – in advance of any formal analysis, for understanding the analysis, and for presenting analysis results.

A random sample is a set of values drawn independently from a larger population. A (uniform) random sample has the characteristic that all members of the population have an equal chance of being drawn. In the previous chapter, we discussed the implications of drawing repeated random samples from a normally distributed population, where the probability that a value lies in a given interval is governed by the normal density. This chapter will expand upon that discussion by using the idea of a sampling distribution, with its associated standard error, to assess estimation accuracy. Confidence intervals and tests of hypotheses offer a formal basis for inference, based on the sampling distribution. We will comment on weaknesses in the hypothesis testing framework.

Basic concepts of estimation

This section will introduce material that is fundamental to inference.

Population parameters and sample statistics

Parameters, such as the mean (μ) or standard deviation (σ), numerically summarize various aspects of a population. Such parameters are usually unknown and are estimated using statistics calculated using a random sample taken from the population. The sample mean is an example of a statistic, and it is used to estimate the population mean.

Other commonly used statistics are the proportion, standard deviation, variance, median, the quartiles, the slope of a regression line, and the correlation coefficient. Each may be used as an estimate of the corresponding population parameter.

Data for which the models of this chapter may be appropriate can be displayed as a scatterplot. The focus will be on the straight line model, though the use of transformations makes it possible to accommodate specific forms of non-linear relationship within this framework. By convention, the x-variable, plotted on the horizontal axis, has the role of explanatory variable. The y-variable, plotted on the vertical axis, has the role of response or outcome variable.

Many of the issues that arise for these simple regression models are fundamental to any study of regression methods. Various special applications of linear regression raise their own specific issues. One such special application, discussed in Subsection 5.6.2, is to size and shape data.

Scrutiny of the scatterplot should precede regression calculations. Such a plot may indicate that the intended regression is plausible, or it may reveal unexpected features.

If there are many observations, it is often useful to compare the fitted line with a fitted smooth curve. If this differs substantially from an intended line, then straight line regression may be inappropriate, as in Figure 2.6. The fitting of such smooth curves will be a major focus of Chapter 7.

Fitting a line to data

How accurate is the line?

Application of the summary function to an lm object from a straight line regression, as in Subsection 5.1.1 following, gives a standard error for each of a and b. Standard errors of predicted values may also or alternatively be of interest; we defer discussion of these until Section 5.3. Determination of these standard errors requires the specific statistical assumptions that will now be noted.

Graphical user interfaces to R

The R Commander (Rcmdr) will be the main focus, with brief reference to other GUIs. Subsection 1.1.1 mentioned the usefulness of the R Commander for data input. Especially for novices or infrequent users of R, a GUI can be similarly helpful for creating simple graphs, for statistical testing, for simple tabulation and summarization, and for fitting standard models. The final subsection describes how the function gui() (fgui package) can be used to create simple GUIs.

Some tasks are best done from the command line, and some from a GUI, with the balance likely to change in favor of the command line as familiarity with R increases. The two modes of use can be mixed. All the GUIs discussed here make available the commands used by R, for inspection and/or modification and/or for audit trail purposes. The user can examine the help page for the relevant function(s), modify the code as required, and re-execute it.

In addition to the R Commander, note:

1. • JGR (Java Graphics for R) and the Deducer GUI that is designed to work with JGR.

2. • The rattle GUI gives access to a range of multivariate graphics, regression and classification routines.

3. • A more limited alternative to the R Commander is pmg (Poor Man's GUI).

4. […]

This chapter starts with comments on the graphical devices that are available in R, on font families and fonts, on plotting symbols, and on R's color choices. It discusses the abilities in the lattice package in modest detail, with a more cursory discussion of the ggplot2 package.

Hardcopy graphics devices

The following writes the graph to the pdf file fossilfuel.pdf:

pdf(file=“fossilfuel.pdf”, width=6.5, height=6.5, pointsize=18)

# For pdf() and postscript(), heights and widths are in inches plot(carbon ˜ year, data=fossilfuel, pch=16) dev.off()

Other functions that open hardcopy graphics devices include png(), jpeg(), bmp(), and tiff(). Unless the default units=“px” is changed, height and width are in pixels. Use dev.off() to close the device, thus making the file available for display, or for printing, or for incorporation into a document. For a complete list of devices, and further details of specific devices, see help(Devices).

Subsection 1.5.4 described commands used to open graphics windows on commonly used implementations of R. Use dev.copy() to copy a graph from the display that is currently active to a hardcopy graphics device. For example:

plot(carbon ˜ year, data=fossilfuel, pch=16) # Display graph ## Now open pdf device and copy graph to it

dev.copy(pdf, file=“fossilcopy.pdf”)

Dimension reduction techniques reduce the number of candidate explanatory variables. Perhaps best known is the replacement of a large number of candidate explanatory variables by the first few principal components. The hope is that they will adequately summarize the information in the candidate explanatory variables. In favorable circumstances, simple modifications of the components will give new variables that are readily interpretable, but this is not always the case.

Propensity scores, often simply called propensities, may be helpful where a response is compared between two groups – a control and a treatment group – that have not been assigned randomly. The response may for example, in a medical context, be death rate in some interval of time. Variables that are not of direct interest, but which may in part explain any differences between the two groups, are commonly known as explanatory variables. Results from such analyses are likely to be suggestive rather than definitive, irrespective of the methodology used to account for explanatory variable effects.

Propensities aim to capture, in a single variable, the explanatory variable effects that are important in accounting for differences between two groups. The propensity score, commonly derived from a discriminant analysis, then becomes the only explanatory variable in the regression calculation.

A time series is a sequence of observations that have been recorded over time. Almost invariably, observations that are close together in time are more strongly correlated than observations that are widely separated. The independence assumption of previous chapters is, in general, no longer valid. Variation in a single spatial dimension may have characteristics akin to those of time series, and the same types of models may find application there also.

Many techniques have been developed to deal with the special nature of the dependence that is commonly found in such series. The present treatment will be introductory and restricted in scope, focusing on autoregressive integrated moving average (ARIMA) models.

In the section that follows, annual depth measurements at a specific site on Lake Huron will be modeled directly as an ARIMA process. Section 9.2 will model a regression where the error term has a time series structure. The chapter will close with a brief discussion of “non-linear” time series, such as have been widely used in modeling financial time series.

The analyses will use functions in the stats package, which is a recommended package, included with binary distributions. Additionally, there will be use of the forecast package. In order to make this available, install the forecasting bundle of packages.

The brief discussion of non-linear time series (ARCH and GARCH models) will require access to the tseries package, which must be installed.

Tree-based methods, or decision tree methods, may be used for two broad types of problem–classification and regression. These methods may be appropriate when there are extensive data, and there is uncertainty about the form in which explanatory variables ought to enter into the model. They may be useful for initial data exploration. Tree-based methods have been especially popular in the data mining community.

Tree-structured classification has a long history in biology, where informal methods of dendrogram construction have been in use for centuries. Social scientists began automating tree-based procedures for classification in the 1940s and 1950s, using methods which are similar to some of the current partitioning methods; see Belson (1959). Venables and Ripley (2002, Chapter 9) give a short survey of the more recent history.

The tree-based regression and classification methodology is radically different from the methods discussed thus far in this book. The theory that underlies the methods of earlier chapters has limited relevance to tree-based methods. The methodology is relatively easy to use and can be applied to a wide class of problems. It is at the same time insensitive to the nuances of particular problems to which it may be applied.

The methodology makes limited use of the ordering of values of continuous or ordinal explanatory variables. In small data sets, it is unlikely to reveal data structure. Its strength is that, in large data sets, it has the potential to reflect relatively complex forms of structure, of a kind that may be hard to detect with conventional regression modeling.

Many regularities of nature are taken for granted in daily living – the rising and setting of the sun, the effects of fire in burning anyone unfortunate enough to get too near, and so on. Experience of the world, rather than logical deductive argument, has identified these regularities. Scientific investigation, especially in the physical sciences, has greatly extended and systematized awareness of regularities. Mathematical descriptions, i.e., models, have been crucial for describing and quantifying these regularities.

As when any model is pressed into service, it is important to understand which features generalize and which do not. An engineer's scale model of a building may be helpful for checking the routing of the plumbing but may give little indication of the acoustics of seminar rooms that are included in the building. In medical research, mouse responses to disease and to therapeutic agents are widely used as models for human responses. Experimental responses in the mouse may indicate likely responses in humans.

In fundamental research in the physical sciences, deterministic models are often adequate. Statistical variability may be so small that it can, for practical purposes, be ignored. In applications of the physical sciences, variability may more commonly be a serious issue. In studying how buildings respond to a demolition charge, there will be variation from one occasion to another, even for identical buildings and identically placed charges. There will be variation in which parts of the building break first, in what parts remain intact, and in the trajectories of fragments.