In the previous chapter, we focused on opinion formation and the various factors that influence that process. In this chapter and the next, we look at what happens after you have formed an opinion. Do you share it with others? And, if so, why do you share it and what specifically do you say?

By “opinion expression,” we refer to an individual’s decision to communicate his or her opinions to others. This decision stage serves as a filter between an individual’s underlying opinions and the sharing of those opinions. As with any filter, not everything that encounters it is going to pass through. An individual may decide not to share any opinion at all or to voice a modified (or moderated) version of his or her actual opinion. In other words, the opinion expression decision can be broken down into two components: (1) whether to share an opinion and (2) what opinion to share.

Imagine you are chatting with some new friends, co-workers, or neighbors whom you have met fairly recently. In such unfamiliar situations, most people err on the side of caution and steer the conversation to safe topics like the weather, local restaurants, new movies , or weekend plans. That way, you don’t run much risk of inadvertently offending the people whom you just met by expressing a potentially contentious opinion.

Before we jump into how organizations can build their social media intelligence capabilities, we first need to understand the science of opinions. Behind every social media comment posted is a person with an opinion. However, not everyone with an opinion chooses to share that opinion online. The opinions we see posted to social media are an outcome of a two-stage process. In the first stage, we form our opinions (opinion formation). Then, in the second stage, we share our opinions with others (opinion expression). However, we don’t share all of our opinions. Instead, the opinions that we ultimately post online are those that somehow merit sharing. In other words, the opinion expression stage can be thought of as an opinion filter that allows some of our opinions to pass through to be posted on social media while our other opinions remain unshared (see Figure 2.1).

To illustrate the distinction between opinion formation and expression, here’s an offline example that most of us are familiar with: the Monday morning quarterback. We all know one, and chances are we know more than one. The Monday morning quarterback has well-developed opinions about Sunday’s football games and wants to share those opinions with anyone willing to listen: Which plays should have been run. Which players should have been substituted. Which calls were blown by the officials and should have been challenged.

As mentioned previously, experimental methods can be a matter of dispute: therecan be as many views of the “correct” way to run an experiment asthere are experimenters. Such disagreements are most obvious in the approachtaken to statistical analysis of data: everyone has their own favourite method,there can be many different valid ways to analyse data, and even statisticiansdo not always agree on the best approach.

This chapter is not intended to be a statistics primer: it simply describes thestatistics tests that I find most useful in analysing data and shows examples oftheir application. It does not discuss any theoretical aspects of these tests orwhy they “work.” Rather, it is a practical guide that will enablean experimenter to make considerable headway with some simple analyses, and tobe able to consult a statistics text for more information with confidence.

In most cases, these tests will be sufficient for answering the type of researchquestions discussed so far. Other analyses may require reference to a goodstatistics book or guidance from a statistics consultant.

This book describes the process that takes a researcher from identifying ahuman–computer interaction (HCI) research idea that needs to be tested,to designing and conducting a test, and then analysing and reporting theresults. This first chapter introduces the notion of an “HCI idea”and different approaches to testing.

Assessing the worth of an HCI idea

Imagine that you have an HCI idea, for example, a novel interaction method, a newway of visualising data, an innovative device for moving a cursor, or a newinteractive system for building games. You can implement it, demonstrate it to awide range of people, and even deploy it for use – but is it a“good” idea? Will the interaction method assist users with theirtasks? Will the visualisation make it easier to spot data trends? Will the newdevice make cursor movement quicker? Will users like the new game buildingsystem?

It is your idea, so of course you believe that it is wonderful; however, yoursubjective judgement (or even the views of your friends in the researchlaboratory) is not sufficient to prove its general worth. An objectiveevaluation of the idea (using people not involved in the research) is required.As Zhai (2003) says in his controversial article, “Evaluation is theworst form of HCI research except all those other forms that have beentried,” the true value of the idea cannot be determined simply by“subjective opinion, authority, intimidation, fashion or fad.”

The definition of the conditions, tasks, and experimental objects is the initialfocus of the experimental design, and must be carefully related to the researchquestion, as described in Chapter 2. The experiment itself could be describedsimply as presenting the stimuli to human participants and asking them toperform the tasks. There are, however, still many other decisions to be madeabout the experimental process, as well as additional supporting materials andprocesses to be considered.

This chapter focuses on the nature of the participant experience, that is, whateach participant will do between the start and end times of the experiment– a lot more happens than simply presenting the trials.

Allocating participants to conditions

As highlighted in Chapter 1, the key issue when running experiments is thecomparison of performance between the conditions: does one condition producebetter or worse performance than another? To determine “performance witha condition,” human participants will need to perform tasks associatedwith the HCI idea being investigated, and measurements of the overallperformance for each condition will be taken. Recall that we want to producedata like that in Figure 2.1, which summarises performance according to eachexperimental condition, with no explicit reference to tasks or experimentalobjects.

To illustrate the two approaches to factor analysis, consider awithin-participant experiment that aims to answer the research question,“Which visual form of an image best supports visual search?” Theindependent variable is the visual form of an image with three conditions: Blackand White (BW), Colour (C), and Grey-scale (GS).

Each screen presents forty items, and there is only one task – identifythe largest image. To ensure generalisability of the results, there are threeexperimental objects, each using a different type of image: images of theenvironment (photographs, P), paintings (photographs of paintings, PP), andgraphics (images created using a digital imaging tool, G). Error and responsetime data are collected, but only error data are analysed here. Data for thisexperiment (fabricated for the purposes of illustration) are shown in TableA3.1.

The primary independent variable primary independent variable isvisual form (BW, C, GS) because this is directly related to the researchquestion. A secondary independent variable is image type (withthree secondary conditions, P, PP, G).

To conclude, this chapter summarises the contents of this book by presenting amodel and six key principles for designing and conducting experiments.

A model of the experimental process

The model, presented in Figure 8.1, shows the main stages of the experimentalprocess and the important considerations that need to be addressed at eachstage.

Six key principles for conducting experiments

This book presents specific advice to guide the researcher through theexperimental process, and, subsequently, six key general principles emerge.These are listed as follows:

The first step in running an experiment is defining what you want to discover and how you will do so. This chapter presents an approach to experiments that begins by first defining a research question, and then basing the definition of the conditions, experimental objects, and tasks on that question. These elements will ultimately define the form of the experiment.

Several key concepts used throughout the book are introduced and defined in this chapter:

• The research question : a clear question that succinctly states the aim of the research;

• Conditions: the ideas of interest – these will be compared against each other;

• The independent variable: the set of conditions to be used in the experiment – there will always be more than one condition;

• The population : all the people who might use the idea; the sample: the set of people who will take part in the experiment;

• Generalisability : the extent to which experimental results can apply to situations not explicitly included in the experiment itself;

• Experimental objects : the way in which the ideas are presented to the participants – experimental objects embody the conditions so that they can be perceived;

• Experimental stimulus : the combination of an experimental object and a condition;

• Experimental tasks : what the participants will actually do with the experimental objects;

• Experimental trial : the combination of a condition, an experimental object, and a task.

Designing an experiment is more than creating stimuli and tasks and deciding onthe participant experience. Before conducting the experiment, the exact form ofdata to be collected needs to be decided, and importantly, it needs to beconfirmed as sufficient for answering the research question.

This chapter focusses primarily on data collection. It describes the differenttypes of data that can be collected for different purposes and the means ofcollecting it.

We make the traditional distinction between quantitative data(represented by numbers; e.g., the number of errors, a preference ranking) andqualitative data (not represented by numbers; e.g., averbal description of problems encountered in performing the task, a videoshowing interaction with an interface).

In practise, there are two distinct decisions to be made about data:

• What data to collect (a decision made in advance of the experiment),and

• How to analyse the data (a decision made after the experiment hasbeen run).

These two decisions are inextricably linked because the potential means ofanalysis will influence the decision on what data to collect. Any discussionabout data collection therefore necessarily entails discussion on how it will beanalysed.

Some years ago, I presented a retrospective of the graph drawing (and related)experiments I had conducted since 1995 to an audience of information visualisationresearchers, describing the process I went through in defining a newexperimental research area and learning to run human–computer interactionexperiments. This was an honest and reflective seminar in which I highlightedthe mistakes I had made, the good and bad decisions, and how my knowledgeof experimental design had increased and improved with every experiment. Atthe end of my presentation, a member of the audience asked, “So, Helen, whatis the ‘Black Art’? What is it that you have learned about running experimentsthat we should all know?”

This started me thinking about how much expertise is embodied in experienceand seldom communicated apart from in a master/apprentice model.PhD supervisors can advise students on how to formulate and conduct experiments,psychology and HCI research texts can be read, and other experimentsin the research literature can be copied, but the actual step-by-step process ofdesigning and running an experiment is rarely written down and communicatedwidely. Although I believe that one can never understand the process of conductingexperiments without experiencing the process oneself, I also believethat experiences can (and should) be shared and that advice resulting fromothers’ experiences can always be useful.

So, now you have your results, and you want to tell everyone about them. This chapter discusses the way in which you report your research, typically in a research article for an academic conference or journal, or in a dissertation for assessment. In all cases, you need to keep in mind that someone else will be reading what you write, and that this person has not been party to your decision-making process. It is easy to leave information out because it appears obvious.

Reviewers’ concerns

It is the job of reviewers or assessors to make a judgement on the worth of your research, and it is your job to make sure that you have presented it sufficiently well that they can do so. This is true of all research; however, writing experimental papers brings with it its own particular issues:

• An experiment focuses on addressing specific research questions: reviewers may not believe that these questions are interesting or important.

• An experimental research question could be addressed in many different ways: everyone will have their own idea as to how best to address it.

• Different statistical methods are favoured by different people: beware the reviewers who are well versed in statistics when you have not analysed your data using their favoured method!

• No experiment can ever be perfect: reviewers can easily find faults on which to base a negative judgement if they want.

• Many reviewers have never actually designed and conducted an experiment themselves: they do not always appreciate the amount of work required for running experiments, the difficulty in making appropriate design decisions, or the constraints that apply to experimental design.