The previous chapter (Chapter 11) explained how user requirements directed our development of meeting support technology, more specifically meeting browsers and assistants. Chapters 3 to 9 discussed the enabling components, i.e. the multimodal signal processing necessary to build meeting support technology. In the following, we will present an overview of the meeting browsers and assistants developed both in AMI and related projects, as well as outside this consortium.

Introduction

Face-to-face meetings are a key method by which organizations create and share knowledge, and the last 20 years have seen the development of new computational technology to support them.

Early research on meeting support technology focused on group decision support systems (Poole and DeSanctis, 1989), and on shared whiteboards and large displays to promote richer forms of collaboration (Mantei, 1988, Moran et al., 1998, Olson et al., 1992, Whittaker and Schwarz, 1995, Whittaker et al., 1999). There were also attempts at devising methods for evaluating these systems (Olson et al., 1992). Subsequent research was inspired by ubiquitous computing (Streitz et al., 1998, Yu et al., 2000), focusing on direct integration of collaborative computing into existing work practices and artifacts. While much of this prior work has addressed support for real-time collaboration by providing richer interaction resources, another important research area is interaction capture and retrieval.

Interaction capture and retrieval is motivated by the observation that much valuable information exchanged in workplace interactions is never recorded, leading people to forget key decisions or repeat prior discussions.

Introduction

While the meeting setting creates many challenges just in terms of recognizing words and who is speaking them, once we have the words, there is still much to be done if the goal is to be able to understand the conversation. To do this, we need to be able to understand the language and the structure of the language being used.

The structure of language is multilayered. At a fine-grained, detailed level, we can look at the structure of the spoken utterances themselves. Dialogue acts which segment and label the utterances into units with one core intention are one type of structure at this level. Another way of looking at understanding language at this level is by focusing on the subjective language being used to express internal mental states, such as opinions, (dis-)agreement, sentiments, and uncertainty.

At a coarser level, language can be structured by the topic of conversation. Finally, within a given topic, there is a structure to the language used to make decisions. Language understanding is sufficiently advanced to capture the content of the conversation for specific phenomena like decisions based on elaborate domain models. This allows an indexing and summarization of meetings at a very high degree of understanding.

Finally, the language of spoken conversation differs significantly from written language. Frequent types of speech disfluencies can be detected and removed with techniques similar to those used for understanding language structure as described above.

Introduction

Segmenting multi-party conversations into homogeneous speaker regions is a fundamental step towards automatic understanding of meetings. This information is used for multiple purposes as adaptation for speaker and speech recognition, as a meta-data extraction tool to navigate meetings, and also as input for automatic interaction analysis.

This task is referred to as speaker diarization and aims at inferring “who spoke when” in an audio stream involving two simultaneous goals: (1) the estimation of the number of speakers in an audio stream and (2) associating each speech segment with a speaker.

Diarization algorithms have been developed extensively for broadcast data, characterized by regular speaker turns, prompted speech, and high-quality audio, while processing meeting recordings presents different needs and additional challenges. From one side, the conversational nature of the speech involves very short turns and large amounts of overlapping speech; from the other side, the audio is acquired in a nonintrusive way using far-field microphones and is thus corrupted with ambient noise and reverberation. Furthermore real-time and online processing are often required in order to enable the use of many applications while the meeting is actually going on. The next section briefly reviews the state-of-the-art in the field.

State of the art in speaker diarization

Conventional speaker diarization systems are composed of the following steps: a feature extraction module that extracts acoustic features like mel-frequency cepstral coefficients (MFCCs) from the audio stream, a Speech/Non-speech Detection which extracts only the speech regions discarding silence, an optional speaker change module which divides the input stream into small homogeneous segments uttered by a single speaker, and an agglomerative hierarchical clustering step which groups together those speech segments into the same cluster.

The basic modeling problem begins with a set of observed data yn = {yt : t = 1, 2, …, n}, generated by some physical machinery, where the elements yt may be of any kind. Since no matter what they are they can be encoded as numbers we take them as such, i.e. natural numbers with or without the order if the data come from finite or countable sets, and real numbers otherwise. Often each number yt is observed together with others x1,t, x2,t, …, called explanatory data, written collectively as a K × n matrix X = {xi,j}, and the data then are written as yn ∣X. It is convenient to use the terminology “variables” for the source of these data. Hence, we say that the data {yt} come from the variable Y, and the explanatory data are generated by variables X1, X2, and so on.

In physics the explanatory data often determine the data yn of interest, called a “law,” but not so in statistical problems. Although by taking sufficiently many explanatory data we may also fit a function to the given set of observed data, but this is not a “law,” since if the same machinery were to generate additional data yn+1, x1,n+1, x2,n+1, … the function would not give yn+1. This is the reason the objective is to learn the statistical properties of the data yn, possibly in the context of the explanatory data.

The 1918 flu pandemic, also referred to as the Spanish flu, was a devastating infectious disease. It is estimated that 50 million people, about 3% of the world's population at the time, died of the disease. About 500 million people were infected. The causative agent was an influenza virus. In this chapter we will learn more about these viruses. We will make use of highly significant molecular biology databases and bioinformatics tools. These are useful not only for learning about influenza viruses, but are widely used to explore just about any topic in biology.

Short history of sequence databases

A vast amount of information is collected by projects around the world designed to characterize genomes, genes and proteins. The development with respect to DNA sequencing is particularly remarkable. One important task in bioinformatics is to store all of this information in databases and, importantly, to make it available to the scientific community for downloading and analysis. Numerous dedicated individuals working on database projects are the unsung heroes of bioinformatics and molecular biology (see also the quotation on stamp collecting above).

Many kinds of information technology can be used to make meetings more productive, some of which are related to what happens before and after meetings, while others are intended to be used during a meeting. Document repositories, presentation software, and even intelligent lighting can all play their part. However, the following discussion of user requirements will be restricted to systems that draw on the multimodal signal processing techniques described in the earlier chapters of this book to capture and analyze meetings. Such systems might help people understand something about a past meeting that has been stored in an archive, or they might aid meeting participants in some way during the meeting itself. For instance, they might help users understand what has been said at a meeting, or even convey an idea of who was present, who spoke, and what the interaction was like. We will refer to all such systems, regardless of their purpose or when they are used, as “meeting support technology.”

This chapter reviews the main methods and studies that elicited and analyzed user needs for meeting support technology in the past decade. The chapter starts by arguing that what is required is an iterative software process that through interaction between developers and potential users gradually narrows and refines sets of requirements for individual applications. Then, it both illustrates the approach and lays out specific user requirements by discussing the major user studies that have been conducted for meeting support technology.

General overview

Meetings are a rich resource of information that, in practice, is mostly untouched by any form of information processing. Even now it is rare that meetings are recorded, and fewer are then annotated for access purposes. Examples of the latter only include meetings held in parliaments, courts, hospitals, banks, etc., where a record is required for reasons of decision tracking or legal obligations. In these cases a labor-intensive manual transcription of the spoken words is produced. Giving much wider access to the rich content is the main aim of the AMI consortium projects, and there are now many examples of interest in that access – through the release of commercial hardware and software services. Especially with the advent of high-quality telephone and videoconferencing systems the opportunity to record, process, recognize, and categorize the interactions in meetings is recognized even by skeptics of speech and language processing technology.

Of course meetings are an audio-visual experience by nature and humans make extensive use of visual and other sensory information. To illustrate the rich landscape of information is the purpose of this book and many applications can be implemented even without looking at the spoken word. However, it is still verbal communication that forms the backbone of most meetings, and accounts for the bulk of the information transferred between participants. Hence automatic speech recognition (ASR) is key to access the information exchanged and is the most important part required for most higher level processing.