As noted early in Chapter 5, point-by-point agreement is often accepted as evidence that summary measures derived from sequential data will be reliable, probably because point-by-point agreement seems the more stringent approach. Kappa-based demonstrations of observer agreement are essential for observer training and for checking throughout coding; but once data collection is complete and summary measures have been derived from the sequential data, their reliability can be assessed directly with an intraclass correlation coefficient (ICC).

Relative Versus Absolute Agreement

Summary measures in observational research are typically of two kinds: first are frequencies, rates, and proportions for individual codes (see Chapter 8); and second are contingency indices and other contingency table statistics involving two or more codes (see Chapter 9). Both are continuous (i.e., measured on an interval or ratio scale), and so an ICC measure of reliability, which requires continuous or arguably ordinal measurement, is appropriate. The general question is: When values are derived from records of the same events coded independently by different observers, do the observers agree sufficiently for our purposes? Can we regard the observers as essentially interchangeable? In this regard, one important distinction concerns relative, as opposed to absolute, judgment (i.e., relative consistency vs. absolute agreement, or norm-referenced vs. criterion-referenced; see Suen, 1988). Do we want scores derived from two observers’ records to be rank-ordered the same way – that is, to be in relative agreement – or do we want them to agree regarding absolute magnitude?