Knowledge Management, Databases and Data Mining

Summary

As discussed in the previous chapter, visualization when applied to text seems to be most effective for specialists doing data analysis. Although this is an exciting field, it is not what most people think of when one talks about search interfaces. Unfortunately, some researchers working on visualization of text conflate search tasks with data analysis tasks. For example, Veerasamy and Heikes (1997) critique one interface for making it “more difficult than in our tool to gain an overall picture of the query word distribution for a whole set of documents in one glance.” It is unclear why a searcher would want to see such a distribution, even though such a view may be of great interest to a computational linguist.

This chapter describes ideas that have been put forward for understanding the contents of text collections from a more analytical point of view. The first section discusses applications in the field of Text Mining, which usually involve visualizing connections among entities within and across documents. The next section discusses methods for visualization occurrences of words or phrases within documents, in what have classically been called concordances. The final section discusses various attempts to visualize relationships between words in their usage in language and in lexical ontologies.

Visualization for Text Mining

There is great interest in the field of text mining, which can be defined as the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources (Feldman and Sanger 2006; Hearst 1999a).

Summary

Currently most Web search engines produce the same results independently of who the user is; search engines are in fact designed to do well at meeting the average goals of the group over the specific goals of the individual (Teevan et al. 2005a). Nonetheless, researchers, marketers, and search pundits alike often state that search results should reflect the fact that people differ in their needs, goals, and preferences (Pitkow et al. 2002). As a typical example, an avionics expert should see a different results ranking for a query on jets than a football fan from New York.

Tailoring search results to an individual's interests is just one form of information personalization. The term is currently applied to a diverse array of information applications. Given a profile of a user's interests, information content can be personalized by: re-ranking search results listings to better reflect the user's preferences, automatically notifying a user when new articles of interest appear or important changes are made to existing documents, recommending items that are related in some way to an item currently being viewed, and customizing the look and content of a Web site or other information artifact.

Information personalization approaches vary along several dimensions. Some make use of information that users explicitly specify about themselves and their interests, while others attempt to infer users' preferences implicitly from their actions. Some approaches note that individuals tend to have both long-term, stable interests, and short-term temporary interests, and that these can come into conflict when proposing recommendations based on past actions.

Summary

Chapter 3 discussed the various theories associated with the information seeking process, and the subsequent chapters described interfaces for supporting query specification, results presentation, and query reformulation – the standard stages of the search process. This chapter describes interface ideas for other aspects of the search process: search starting points, search history, and interfaces that support the process as a whole. The final section discusses attempts to integrate search into the sensemaking process.

Starting Points for Search

The first step in addressing an information need is deciding which tools to use and which collections to search over, a process which is sometimes referred to as source selection. Today there are many choices, including phoning, emailing, or texting a friend, reaching for a physical book, going to a physical library, or sitting down at a networked computer and starting up a Web browser.

Starting Points in Web Search

For those who go online, today the most common starting point is to open a Web browser and start with a Web search engine. Today, Web browsers make that choice even easier by including an always-visible entry form in the browser's “chrome” or by supporting search directly in what used to be the address entry form. But people also commonly start searches from favorite information resources, such as bookmarked Web sites (Teevan et al. 2004). Web browsers have always allowed users to retain bookmarks, but today are making site revisitation even easier by showing usage history as a drop-down menu within the address bar, and matching that history as the user types.

Summary

In order to design successful search user interfaces, it is necessary to understand the human information seeking process, including the strategies people employ when engaged in search. Numerous theoretical treatments have been proposed to characterize this complex cognitive process (Belkin et al. 1982; Jarvelin and Ingwersen 2004; Kuhlthau 1991; Marchionini 1995; Saracevic 1997; Sutcliffe and Ennis's 1998). This chapter presents the most commonly discussed theoretical models of the search process: the standard model, the cognitive model, the dynamic model, search as a sequence of stages, search as a strategic process, and sensemaking. The chapter concludes with a discussion of information needs, including methods for inferring information needs from their expression as queries.

The Standard Model of Information Seeking

Many accounts of the information seeking process assume an interaction cycle consisting of identifying an information need, followed by the activities of query specification, examination of retrieval results, and if needed, reformulation of the query, repeating the cycle until a satisfactory result set is found (Salton 1989; Shneiderman et al. 1998). As Marchionini (1989) puts it:

“Information-seeking is a special case of problem solving. It includes recognizing and interpreting the information problem, establishing a plan of search, conducting the search, evaluating the results, and if necessary, iterating through the process again.”

This model is elaborated by Sutcliffe and Ennis's (1998) oft-cited information seeking process model, which they formulate as a cycle consisting of four main activities:

Problem identification,
Articulation of information need(s),
Query formulation, and
Results evaluation.

Summary

For images appearing in Figures 8.8, 8.9, 8.10, and 8.11, the following credit information applies:

T. Tokuhara (Japanese, active late 19th century)

House on Lake with Boy and Man in Foreground, ca. 1890–1900

Albumen silver print with hand coloring

21 × 27.1 cm (image)

Fine Arts Museums of San Francisco, Gift of James H. Soong, 1989.3.25

Hiroshige (Japanese, 1797–1858)

At Yase Village (Yase no sato), from the series Famous Places in Kyoto

(Kyoto meisho no uchi), ca. 1833–1834

Color woodcut

21.9 × 35.5 cm (sheet)

Fine Arts Museums of San Francisco, Gift of Jane Scribner, 1982.1.92

Eisen (Japanese, 1790–1848)

The Mimeguri Embankment of the Sumida River, ca. 1815–1820

Color woodcut surimono

17.6 × 47.1 cm (sheet)

Fine Arts Museums of San Francisco, Gift of Katherine Ball to the DeYoung

Museum, transferred to AFGA, 1964.141.970

Eishi (Japanese, 1756–1829)

The Courtesan Takigawa of the Tamaya with Attendants (a Shinzo

and Her Two Kamuro, Chidori and Namiji) (Tamaya no uchi Takigawa,

Chidori, Namiji), ca. 1788–1790

Color woodcut

37.1 × 25.6 cm (image)

Fine Arts Museums of San Francisco, Gift of Miss Carlotta Mabury, 54755.32

Eishi (Japanese, 1756–1829)

Kneeling Woman Showing a Miniature Plum Tree to a Child in Its Mother's

Artms, from the series Children's Pastimes in the Four Seasons (Shiki

warabe asobi), ca. 1794–1796

Summary

In the query specification part of the information access process, the searcher expresses an information need by converting their internalized, abstract concepts into language, and then converting that expression of language into a query format that the search system can make use of. This chapter discusses the mechanisms by which information needs are expressed. The two main dimensions for the query specification process are:

(1) The kind of information the searcher supplies. Query specification input spans a spectrum from full natural language sentences, to keywords and key phrases, to syntax-heavy command language-based queries.
(2) The interface mechanism the user interacts with to supply this information. These include command line interfaces, graphical entry form based interfaces, and interfaces for navigating links.

Each is discussed in the sections below.

Textual Query Specification

Queries over collections of textual information usually take on a textual form (querying against multimedia is discussed in Chapter 12). The next subsections discuss different kinds of textual input for query specification.

Search Over Surrogates vs. Full Text

In older bibliographic search systems, users could only search over metadata that hinted at the underlying contents. To look for a book on a topic in an online library catalog, the searcher was restricted to the text in the title or the few subject labels that the librarian who had catalogued the book had used to describe it (Borgman 1996). A book that mentions an interesting idea, but only in a secondary manner, would most likely not be indexed with that term (Cousins 1992).

Summary

Search is an integral part of peoples’ online lives; people turn to search engines for help with a wide range of needs and desires, from satisfying idle curiousity to finding life-saving health remedies, from learning about medieval art history to finding video game solutions and pop music lyrics. Web search engines are now the second most frequently used online computer application, after email. Not long ago, most software applications did not contain a search module. Today, search is fully integrated into operating systems and is viewed as an essential part of most information systems.

Many books on information retrieval describe the algorithms behind search engines and information retrieval systems. By contrast, this book focuses on the human users of search systems and the tool they use to interact with them: the search user interface. Because of their global reach, search user interfaces must be understandable by and appealing to a wide variety of people of all ages, cultures, and backgrounds, and for an enormous variety of information needs.

The truly worldwide reach of the Web has brought with it a new realization among computer scientists and laypersons alike of the enormous importance of usability and user interface design. In the last ten years, much has become understood about what works in search interfaces from a usability perspective, and what does not. Researchers and practitioners have developed a wide range of innovative interface ideas, but only the most broadly acceptable make their way into major Web search engines.

Summary

Throughout this book, the merits of an interface are assessed from a usability perspective; this chapter discusses how these assessments are done. It is surprisingly difficult to design a usable new search interface, and perhaps even harder to convincingly assess its usability (Draper and Dunlop 1997). An evaluator must take into account differences in designs, tasks, participant motivation, and knowledge, all of which can vary the outcome of a study. Furthermore, as mentioned in Chapter 1, small details in the design of the interface can have a strong effect on a participant's subjective reaction to or objective success with the interface. For instance, problematic placement of controls or unintuitive text on a hyperlink can prevent a participant from succeeding in a task. Differences in font contrast and spacing can unconsciously affect a participant's subjective response to a design.

What should be measured when assessing a search interface? Traditional information retrieval research focuses on evaluating the proportion of relevant documents retrieved in response to a query. In evaluating search user interfaces, this kind of measure can also be used, but is just one component within broader usability measures. Recall from Chapter 1 that usable interfaces are defined in terms of learnability, efficiency, memorability, error reduction, and user satisfaction (Nielsen 2003b; Shneiderman and Plaisant 2004).

Summary

And as noted in Chapter 3, a common search strategy is for the user to first issue a general query, then look at a few results, and if the desired information is not found, to make changes to the query in an attempt to improve the results. This cycle is repeated until the user is satisfied or gives up. The previous two chapters discussed interfaces for query specification and presentation of search results. This chapter discusses the query reformulation step.

The Need for Reformulation

Examination of search engine query logs suggests a high frequency of query reformulation. One study by Jansen et al. (2005) analyzed 3 million records from a 24 hour snapshot of Web logs taken in 2002 from the AltaVista search engine. (The search activity was partitioned into sessions separated by periods of inactivity, and no effort was made to determine if users searched for more than one topic during a session. 72% of the sessions were less than five minutes long, and so one-topic-per-session is a reasonable, if noisy, estimate.) The analysis found that the proportion of users who modified queries was 52%, with 32% issuing 3 or more queries within the session. Other studies show similar proportions of refinements, thus supporting the assertion that query reformulation is a common part of the search process.

Good tools are needed to aid in the query formulation process. At times, when a searcher chooses a way to express an information need that does not successfully match relevant documents, the searcher becomes reluctant to radically modify their original query and stays stuck on the original formulation.

Text mining is a new and exciting area of computer science research that tries to solve the crisis of information overload by combining techniques from data mining, machine learning, natural language processing, information retrieval, and knowledge management. Similarly, link detection – a rapidly evolving approach to the analysis of text that shares and builds upon many of the key elements of text mining – also provides new tools for people to better leverage their burgeoning textual data resources. The Text Mining Handbook presents a comprehensive discussion of the state-of-the-art in text mining and link detection. In addition to providing an in-depth examination of core text mining and link detection algorithms and operations, the book examines advanced pre-processing techniques, knowledge representation considerations, and visualization approaches. Finally, the book explores current real-world, mission-critical applications of text mining and link detection in such varied fields as M&A business intelligence, genomics research and counter-terrorism activities.

Summary

Recall the major steps in inverted index construction:

Summary

INFORMATION RETRIEVAL

The meaning of the term information retrieval (IR) can be very broad. Just getting a credit card out of your wallet so that you can type in the card number is a form of information retrieval. However, as an academic field of study, information retrieval might be defined thus:

Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).

As defined in this way, information retrieval used to be an activity that only a few people engaged in: reference librarians, paralegals, and similar professional searchers. Now the world has changed, and hundreds of millions of people engage in information retrieval every day when they use a web search engine or search their email. Information retrieval is fast becoming the dominant form of information access, overtaking traditional database-style searching (the sort that is going on when a clerk says to you: “I'm sorry, I can only look up your order if you can give me your order ID”).

Information retrieval can also cover other kinds of data and information problems beyond that specified in the core definition above. The term “unstructured data” refers to data that does not have clear, semantically overt, easy-for-a-computer structure. It is the opposite of structured data, the canonical example of which is a relational database, of the sort companies usually use to maintain product inventories and personnel records.

Summary

On page 113, we introduced the notion of a term-document matrix: an M × N matrix C, each of whose rows represents a term and each of whose columns represents a document in the collection. Even for a collection of modest size, the term-document matrix C is likely to have several tens of thousands of rows and columns. In Section 18.1.1, we first develop a class of operations from linear algebra, known as matrix decomposition. In Section 18.2, we use a special form of matrix decomposition to construct a low-rank approximation to the term-document matrix. In Section 18.3 we examine the application of such low-rank approximations to indexing and retrieving documents, a technique referred to as latent semantic indexing. Although latent semantic indexing has not been established as a significant force in scoring and ranking for information retrieval (IR), it remains an intriguing approach to clustering in a number of domains including for collections of text documents (Section 16.6, page 343). Understanding its full potential remains an area of active research.

Readers who do not require a refresher on linear algebra may skip Section 18.1, although Example 18.1 is especially recommended as it highlights a property of eigenvalues that we exploit later in the chapter.

Linear algebra review

We briefly review some necessary background in linear algebra. Let C be an M × N matrix with real-valued entries; for a term–document matrix, all entries are in fact non-negative.

Summary

Thus far, this book has mainly discussed the process of ad hoc retrieval, where users have transient information needs that they try to address by posing one or more queries to a search engine. However, many users have ongoing information needs. For example, you might need to track developments in multicore computer chips. One way of doing this is to issue the query multicore and computer and chip against an index of recent newswire articles each morning. In this and the following two chapters we examine the question: How can this repetitive task be automated? To this end, many systems support standing queries. A standing query is like any other query except that it is periodically executed on a collection to which new documents are incrementally added over time.

If your standing query is just multicore and computer and chip, you will tend to miss many relevant new articles which use other terms such as multicore processors. To achieve good recall, standing queries thus have to be refined over time and can gradually become quite complex. In this example, using a Boolean search engine with stemming, you might end up with a query like (multicore or multi-core) and (chip or processor or microprocessor).

To capture the generality and scope of the problem space to which standing queries belong, we now introduce the general notion of a classification problem. Given a set of classes, we seek to determine which class(es) a given object belongs to.

Summary

WILDCARD QUERY

In Chapters 1 and 2, we developed the ideas underlying inverted indexes for handling Boolean and proximity queries. Here, we develop techniques that are robust to typographical errors in the query, as well as alternative spellings. In Section 3.1, we develop data structures that help the search for terms in the vocabulary in an inverted index. In Section 3.2, we study the idea of a wildcard query: a query such as *a*e*i*o*u*, which seeks documents containing any term that includes all the five vowels in sequence. The * symbol indicates any (possibly empty) string of characters. Users pose such queries to a search engine when they are uncertain about how to spell a query term, or seek documents containing variants of a query term; for instance, the query automat* seeks documents containing any of the terms automatic, automation, and automated.

We then turn to other forms of imprecisely posed queries, focusing on spelling errors in Section 3.3. Users make spelling errors either by accident, or because the term they are searching for (e.g., Herman) has no unambiguous spelling in the collection. We detail a number of techniques for correcting spelling errors in queries, one term at a time as well as for an entire string of query terms. Finally, in Section 3.4 we study a method for seeking vocabulary terms that are phonetically close to the query term(s).

Summary

Overview

Web crawling is the process by which we gather pages from the Web to index them and support a search engine. The objective of crawling is to quickly and efficiently gather as many useful web pages as possible, together with the link structure that interconnects them. In Chapter 19, we studied the complexities of the Web stemming from its creation by millions of uncoordinated individuals. In this chapter, we study the resulting difficulties for crawling the Web. The focus of this chapter is the component shown in Figure 19.7 as web crawler; it is sometimes referred to as a spider.

The goal of this chapter is not to describe how to build the crawler for a full-scale commercial web search engine. We focus instead on a range of issues that are generic to crawling from the student project scale to substantial research projects. We begin (Section 20.1.1) by listing desiderata for web crawlers, and then discuss in Section 20.2 how each of these issues is addressed. The remainder of this chapter describes the architecture and some implementation details for a distributed web crawler that satisfies these features. Section 20.3 discusses distributing indexes across many machines for a web-scale implementation.

Features a crawler must provide

We list the desiderata for web crawlers in two categories: features that web crawlers must provide, followed by features they should provide.

Knowledge Management, Databases and Data Mining

Refine search

Refine search

Actions for selected content:

1835 results in Knowledge Management, Databases and Data Mining

11 - Information Visualization for Text Analysis

Summary

9 - Personalization in Search

Summary

7 - Supporting the Search Process

Summary

3 - Models of the Information Seeking Process

Summary

Appendix: Additional Copyright Notices

Summary

4 - Query Specification

Summary

Preface

Summary

2 - The Evaluation of Search User Interfaces

Summary

6 - Query Reformulation

Summary

Contents

Author Index

The Text Mining Handbook

2 - The term vocabulary and postings lists

Summary

1 - Boolean retrieval

Summary

Table of Notation

18 - Matrix decompositions and latent semantic indexing

Summary

13 - Text classification and Naive Bayes

Summary

Contents

3 - Dictionaries and tolerant retrieval

Summary

20 - Web crawling and indexes

Summary

Knowledge Management, Databases and Data Mining

Refine search

Refine search

Actions for selected content:

Save Search

1835 results in Knowledge Management, Databases and Data Mining

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

The Text Mining Handbook

Summary

Summary

Summary

Summary

Summary

Summary