To conclude, this chapter summarises the contents of this book by presenting amodel and six key principles for designing and conducting experiments.

A model of the experimental process

The model, presented in Figure 8.1, shows the main stages of the experimentalprocess and the important considerations that need to be addressed at eachstage.

Six key principles for conducting experiments

This book presents specific advice to guide the researcher through theexperimental process, and, subsequently, six key general principles emerge.These are listed as follows:

The first step in running an experiment is defining what you want to discover and how you will do so. This chapter presents an approach to experiments that begins by first defining a research question, and then basing the definition of the conditions, experimental objects, and tasks on that question. These elements will ultimately define the form of the experiment.

Several key concepts used throughout the book are introduced and defined in this chapter:

• The research question : a clear question that succinctly states the aim of the research;

• Conditions: the ideas of interest – these will be compared against each other;

• The independent variable: the set of conditions to be used in the experiment – there will always be more than one condition;

• The population : all the people who might use the idea; the sample: the set of people who will take part in the experiment;

• Generalisability : the extent to which experimental results can apply to situations not explicitly included in the experiment itself;

• Experimental objects : the way in which the ideas are presented to the participants – experimental objects embody the conditions so that they can be perceived;

• Experimental stimulus : the combination of an experimental object and a condition;

• Experimental tasks : what the participants will actually do with the experimental objects;

• Experimental trial : the combination of a condition, an experimental object, and a task.

Designing an experiment is more than creating stimuli and tasks and deciding onthe participant experience. Before conducting the experiment, the exact form ofdata to be collected needs to be decided, and importantly, it needs to beconfirmed as sufficient for answering the research question.

This chapter focusses primarily on data collection. It describes the differenttypes of data that can be collected for different purposes and the means ofcollecting it.

We make the traditional distinction between quantitative data(represented by numbers; e.g., the number of errors, a preference ranking) andqualitative data (not represented by numbers; e.g., averbal description of problems encountered in performing the task, a videoshowing interaction with an interface).

In practise, there are two distinct decisions to be made about data:

• What data to collect (a decision made in advance of the experiment),and

• How to analyse the data (a decision made after the experiment hasbeen run).

These two decisions are inextricably linked because the potential means ofanalysis will influence the decision on what data to collect. Any discussionabout data collection therefore necessarily entails discussion on how it will beanalysed.

Some years ago, I presented a retrospective of the graph drawing (and related)experiments I had conducted since 1995 to an audience of information visualisationresearchers, describing the process I went through in defining a newexperimental research area and learning to run human–computer interactionexperiments. This was an honest and reflective seminar in which I highlightedthe mistakes I had made, the good and bad decisions, and how my knowledgeof experimental design had increased and improved with every experiment. Atthe end of my presentation, a member of the audience asked, “So, Helen, whatis the ‘Black Art’? What is it that you have learned about running experimentsthat we should all know?”

This started me thinking about how much expertise is embodied in experienceand seldom communicated apart from in a master/apprentice model.PhD supervisors can advise students on how to formulate and conduct experiments,psychology and HCI research texts can be read, and other experimentsin the research literature can be copied, but the actual step-by-step process ofdesigning and running an experiment is rarely written down and communicatedwidely. Although I believe that one can never understand the process of conductingexperiments without experiencing the process oneself, I also believethat experiences can (and should) be shared and that advice resulting fromothers’ experiences can always be useful.

So, now you have your results, and you want to tell everyone about them. This chapter discusses the way in which you report your research, typically in a research article for an academic conference or journal, or in a dissertation for assessment. In all cases, you need to keep in mind that someone else will be reading what you write, and that this person has not been party to your decision-making process. It is easy to leave information out because it appears obvious.

Reviewers’ concerns

It is the job of reviewers or assessors to make a judgement on the worth of your research, and it is your job to make sure that you have presented it sufficiently well that they can do so. This is true of all research; however, writing experimental papers brings with it its own particular issues:

• An experiment focuses on addressing specific research questions: reviewers may not believe that these questions are interesting or important.

• An experimental research question could be addressed in many different ways: everyone will have their own idea as to how best to address it.

• Different statistical methods are favoured by different people: beware the reviewers who are well versed in statistics when you have not analysed your data using their favoured method!

• No experiment can ever be perfect: reviewers can easily find faults on which to base a negative judgement if they want.

• Many reviewers have never actually designed and conducted an experiment themselves: they do not always appreciate the amount of work required for running experiments, the difficulty in making appropriate design decisions, or the constraints that apply to experimental design.

So far, we have assumed that everything will go smoothly; however, in practise, this is rarely the case. All will not go as planned, especially if this is your first experiment. Thus, this chapter discusses some of the things that can go wrong, and gives suggestions as to how to prevent them occurring, or how to deal with them if they do. Problems can be prepared for, and in many cases, following the advice given in this and previous chapters will put you in a good position to address them (be forewarned!). Pitfalls are those events for which you cannot prepare but that must be dealt with in order to rescue the situation.

Problems

Pilot tests show that the experimental design is fundamentally flawed. You may have put a great deal of work into preparing the experimental objects, tutorials, etc., only to discover that the task given to the participants is simply too difficult and takes too long, that the participants cannot understand what is expected of them, or that the tasks are actually inappropriate for the different conditions. The concept of pre-pilots (and even pre-pre-pilots!) is useful here. Piloting is an iterative process. Although you must pilot at least once with the full experimental method before running it, it is useful to run smaller, partial pilots on some aspects of the experiment before putting it all together. For example, get feedback on the tutorial from a colleague to find out whether it is clear, or ask someone to perform the tasks on the experimental objects on paper to determine whether they are appropriate. By the time you get to running the final pilots, many of the potential problems will have already been addressed.

Where you will become BFF with GFF

The average life scientist will spend a lot of time working with data. Increasingly, this data will exist in the form of large data sets that will have been downloaded or extracted from one of the many large biological databases that are accessible on the internet. Such in silico data might consist of a small number of very large files, a large number of very small files, or anything in between these extremes. However, in many cases the default file format for those files will be plain text. The actual format of the plain-text file will vary a lot, but the fact that it is plain text means there are many Unix commands that are just waiting to get their hands on your data.

This part of the book will cover a small number of extremely powerful Unix commands that are well suited for slicing and dicing text files. If you are reading this part of the book after working through the ‘Essential Perl’ section then you will spot the similarities between some of these commands and some of the operators in Perl. Conversely, if you have yet to start learning Perl, you will find this section introduces many topics that will be revisited as you learn Perl.