Knowledge Management, Databases and Data Mining

Summary

For images appearing in Figures 8.8, 8.9, 8.10, and 8.11, the following credit information applies:

T. Tokuhara (Japanese, active late 19th century)

House on Lake with Boy and Man in Foreground, ca. 1890–1900

Albumen silver print with hand coloring

21 × 27.1 cm (image)

Fine Arts Museums of San Francisco, Gift of James H. Soong, 1989.3.25

Hiroshige (Japanese, 1797–1858)

At Yase Village (Yase no sato), from the series Famous Places in Kyoto

(Kyoto meisho no uchi), ca. 1833–1834

Color woodcut

21.9 × 35.5 cm (sheet)

Fine Arts Museums of San Francisco, Gift of Jane Scribner, 1982.1.92

Eisen (Japanese, 1790–1848)

The Mimeguri Embankment of the Sumida River, ca. 1815–1820

Color woodcut surimono

17.6 × 47.1 cm (sheet)

Fine Arts Museums of San Francisco, Gift of Katherine Ball to the DeYoung

Museum, transferred to AFGA, 1964.141.970

Eishi (Japanese, 1756–1829)

The Courtesan Takigawa of the Tamaya with Attendants (a Shinzo

and Her Two Kamuro, Chidori and Namiji) (Tamaya no uchi Takigawa,

Chidori, Namiji), ca. 1788–1790

Color woodcut

37.1 × 25.6 cm (image)

Fine Arts Museums of San Francisco, Gift of Miss Carlotta Mabury, 54755.32

Eishi (Japanese, 1756–1829)

Kneeling Woman Showing a Miniature Plum Tree to a Child in Its Mother's

Artms, from the series Children's Pastimes in the Four Seasons (Shiki

warabe asobi), ca. 1794–1796

Summary

In the query specification part of the information access process, the searcher expresses an information need by converting their internalized, abstract concepts into language, and then converting that expression of language into a query format that the search system can make use of. This chapter discusses the mechanisms by which information needs are expressed. The two main dimensions for the query specification process are:

(1) The kind of information the searcher supplies. Query specification input spans a spectrum from full natural language sentences, to keywords and key phrases, to syntax-heavy command language-based queries.
(2) The interface mechanism the user interacts with to supply this information. These include command line interfaces, graphical entry form based interfaces, and interfaces for navigating links.

Each is discussed in the sections below.

Textual Query Specification

Queries over collections of textual information usually take on a textual form (querying against multimedia is discussed in Chapter 12). The next subsections discuss different kinds of textual input for query specification.

Search Over Surrogates vs. Full Text

In older bibliographic search systems, users could only search over metadata that hinted at the underlying contents. To look for a book on a topic in an online library catalog, the searcher was restricted to the text in the title or the few subject labels that the librarian who had catalogued the book had used to describe it (Borgman 1996). A book that mentions an interesting idea, but only in a secondary manner, would most likely not be indexed with that term (Cousins 1992).

Summary

Search is an integral part of peoples’ online lives; people turn to search engines for help with a wide range of needs and desires, from satisfying idle curiousity to finding life-saving health remedies, from learning about medieval art history to finding video game solutions and pop music lyrics. Web search engines are now the second most frequently used online computer application, after email. Not long ago, most software applications did not contain a search module. Today, search is fully integrated into operating systems and is viewed as an essential part of most information systems.

Many books on information retrieval describe the algorithms behind search engines and information retrieval systems. By contrast, this book focuses on the human users of search systems and the tool they use to interact with them: the search user interface. Because of their global reach, search user interfaces must be understandable by and appealing to a wide variety of people of all ages, cultures, and backgrounds, and for an enormous variety of information needs.

The truly worldwide reach of the Web has brought with it a new realization among computer scientists and laypersons alike of the enormous importance of usability and user interface design. In the last ten years, much has become understood about what works in search interfaces from a usability perspective, and what does not. Researchers and practitioners have developed a wide range of innovative interface ideas, but only the most broadly acceptable make their way into major Web search engines.

Summary

Throughout this book, the merits of an interface are assessed from a usability perspective; this chapter discusses how these assessments are done. It is surprisingly difficult to design a usable new search interface, and perhaps even harder to convincingly assess its usability (Draper and Dunlop 1997). An evaluator must take into account differences in designs, tasks, participant motivation, and knowledge, all of which can vary the outcome of a study. Furthermore, as mentioned in Chapter 1, small details in the design of the interface can have a strong effect on a participant's subjective reaction to or objective success with the interface. For instance, problematic placement of controls or unintuitive text on a hyperlink can prevent a participant from succeeding in a task. Differences in font contrast and spacing can unconsciously affect a participant's subjective response to a design.

What should be measured when assessing a search interface? Traditional information retrieval research focuses on evaluating the proportion of relevant documents retrieved in response to a query. In evaluating search user interfaces, this kind of measure can also be used, but is just one component within broader usability measures. Recall from Chapter 1 that usable interfaces are defined in terms of learnability, efficiency, memorability, error reduction, and user satisfaction (Nielsen 2003b; Shneiderman and Plaisant 2004).

Summary

And as noted in Chapter 3, a common search strategy is for the user to first issue a general query, then look at a few results, and if the desired information is not found, to make changes to the query in an attempt to improve the results. This cycle is repeated until the user is satisfied or gives up. The previous two chapters discussed interfaces for query specification and presentation of search results. This chapter discusses the query reformulation step.

The Need for Reformulation

Examination of search engine query logs suggests a high frequency of query reformulation. One study by Jansen et al. (2005) analyzed 3 million records from a 24 hour snapshot of Web logs taken in 2002 from the AltaVista search engine. (The search activity was partitioned into sessions separated by periods of inactivity, and no effort was made to determine if users searched for more than one topic during a session. 72% of the sessions were less than five minutes long, and so one-topic-per-session is a reasonable, if noisy, estimate.) The analysis found that the proportion of users who modified queries was 52%, with 32% issuing 3 or more queries within the session. Other studies show similar proportions of refinements, thus supporting the assertion that query reformulation is a common part of the search process.

Good tools are needed to aid in the query formulation process. At times, when a searcher chooses a way to express an information need that does not successfully match relevant documents, the searcher becomes reluctant to radically modify their original query and stays stuck on the original formulation.

Text mining is a new and exciting area of computer science research that tries to solve the crisis of information overload by combining techniques from data mining, machine learning, natural language processing, information retrieval, and knowledge management. Similarly, link detection – a rapidly evolving approach to the analysis of text that shares and builds upon many of the key elements of text mining – also provides new tools for people to better leverage their burgeoning textual data resources. The Text Mining Handbook presents a comprehensive discussion of the state-of-the-art in text mining and link detection. In addition to providing an in-depth examination of core text mining and link detection algorithms and operations, the book examines advanced pre-processing techniques, knowledge representation considerations, and visualization approaches. Finally, the book explores current real-world, mission-critical applications of text mining and link detection in such varied fields as M&A business intelligence, genomics research and counter-terrorism activities.

Summary

Recall the major steps in inverted index construction:

Summary

INFORMATION RETRIEVAL

The meaning of the term information retrieval (IR) can be very broad. Just getting a credit card out of your wallet so that you can type in the card number is a form of information retrieval. However, as an academic field of study, information retrieval might be defined thus:

Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).

As defined in this way, information retrieval used to be an activity that only a few people engaged in: reference librarians, paralegals, and similar professional searchers. Now the world has changed, and hundreds of millions of people engage in information retrieval every day when they use a web search engine or search their email. Information retrieval is fast becoming the dominant form of information access, overtaking traditional database-style searching (the sort that is going on when a clerk says to you: “I'm sorry, I can only look up your order if you can give me your order ID”).

Information retrieval can also cover other kinds of data and information problems beyond that specified in the core definition above. The term “unstructured data” refers to data that does not have clear, semantically overt, easy-for-a-computer structure. It is the opposite of structured data, the canonical example of which is a relational database, of the sort companies usually use to maintain product inventories and personnel records.

Summary

On page 113, we introduced the notion of a term-document matrix: an M × N matrix C, each of whose rows represents a term and each of whose columns represents a document in the collection. Even for a collection of modest size, the term-document matrix C is likely to have several tens of thousands of rows and columns. In Section 18.1.1, we first develop a class of operations from linear algebra, known as matrix decomposition. In Section 18.2, we use a special form of matrix decomposition to construct a low-rank approximation to the term-document matrix. In Section 18.3 we examine the application of such low-rank approximations to indexing and retrieving documents, a technique referred to as latent semantic indexing. Although latent semantic indexing has not been established as a significant force in scoring and ranking for information retrieval (IR), it remains an intriguing approach to clustering in a number of domains including for collections of text documents (Section 16.6, page 343). Understanding its full potential remains an area of active research.

Readers who do not require a refresher on linear algebra may skip Section 18.1, although Example 18.1 is especially recommended as it highlights a property of eigenvalues that we exploit later in the chapter.

Linear algebra review

We briefly review some necessary background in linear algebra. Let C be an M × N matrix with real-valued entries; for a term–document matrix, all entries are in fact non-negative.

Summary

Thus far, this book has mainly discussed the process of ad hoc retrieval, where users have transient information needs that they try to address by posing one or more queries to a search engine. However, many users have ongoing information needs. For example, you might need to track developments in multicore computer chips. One way of doing this is to issue the query multicore and computer and chip against an index of recent newswire articles each morning. In this and the following two chapters we examine the question: How can this repetitive task be automated? To this end, many systems support standing queries. A standing query is like any other query except that it is periodically executed on a collection to which new documents are incrementally added over time.

If your standing query is just multicore and computer and chip, you will tend to miss many relevant new articles which use other terms such as multicore processors. To achieve good recall, standing queries thus have to be refined over time and can gradually become quite complex. In this example, using a Boolean search engine with stemming, you might end up with a query like (multicore or multi-core) and (chip or processor or microprocessor).

To capture the generality and scope of the problem space to which standing queries belong, we now introduce the general notion of a classification problem. Given a set of classes, we seek to determine which class(es) a given object belongs to.

Summary

WILDCARD QUERY

In Chapters 1 and 2, we developed the ideas underlying inverted indexes for handling Boolean and proximity queries. Here, we develop techniques that are robust to typographical errors in the query, as well as alternative spellings. In Section 3.1, we develop data structures that help the search for terms in the vocabulary in an inverted index. In Section 3.2, we study the idea of a wildcard query: a query such as *a*e*i*o*u*, which seeks documents containing any term that includes all the five vowels in sequence. The * symbol indicates any (possibly empty) string of characters. Users pose such queries to a search engine when they are uncertain about how to spell a query term, or seek documents containing variants of a query term; for instance, the query automat* seeks documents containing any of the terms automatic, automation, and automated.

We then turn to other forms of imprecisely posed queries, focusing on spelling errors in Section 3.3. Users make spelling errors either by accident, or because the term they are searching for (e.g., Herman) has no unambiguous spelling in the collection. We detail a number of techniques for correcting spelling errors in queries, one term at a time as well as for an entire string of query terms. Finally, in Section 3.4 we study a method for seeking vocabulary terms that are phonetically close to the query term(s).

Summary

Overview

Web crawling is the process by which we gather pages from the Web to index them and support a search engine. The objective of crawling is to quickly and efficiently gather as many useful web pages as possible, together with the link structure that interconnects them. In Chapter 19, we studied the complexities of the Web stemming from its creation by millions of uncoordinated individuals. In this chapter, we study the resulting difficulties for crawling the Web. The focus of this chapter is the component shown in Figure 19.7 as web crawler; it is sometimes referred to as a spider.

The goal of this chapter is not to describe how to build the crawler for a full-scale commercial web search engine. We focus instead on a range of issues that are generic to crawling from the student project scale to substantial research projects. We begin (Section 20.1.1) by listing desiderata for web crawlers, and then discuss in Section 20.2 how each of these issues is addressed. The remainder of this chapter describes the architecture and some implementation details for a distributed web crawler that satisfies these features. Section 20.3 discusses distributing indexes across many machines for a web-scale implementation.

Features a crawler must provide

We list the desiderata for web crawlers in two categories: features that web crawlers must provide, followed by features they should provide.

Summary

Thus far, we have dealt with indexes that support Boolean queries: A document either matches or does not match a query. In the case of large document collections, the resulting number of matching documents can far exceed the number a human user could possibly sift through. Accordingly, it is essential for a search engine to rank-order the documents matching a query. To do this, the search engine computes, for each matching document, a score with respect to the query at hand. In this chapter, we initiate the study of assigning a score to a (query, document) pair. This chapter consists of three main ideas.

Summary

Clustering algorithms group a set of documents into subsets or clusters. The algorithms' goal is to create clusters that are coherent internally, but clearly different from each other. In other words, documents within a cluster should be as similar as possible; and documents in one cluster should be as dissimilar as possible from documents in other clusters.

Clustering is the most common form of unsupervised learning. No supervision means that there is no human expert who has assigned documents to classes. In clustering, it is the distribution and makeup of the data that will determine cluster membership. A simple example is Figure 16.1. It is visually clear that there are three distinct clusters of points. This chapter and Chapter 17 introduce algorithms that find such clusters in an unsupervised fashion.

The difference between clustering and classification may not seem great at first. After all, in both cases we have a partition of a set of documents into groups. But as we will see the two problems are fundamentally different. Classification is a form of supervised learning (Chapter 13, page 237): Our goal is to replicate a categorical distinction that a human supervisor imposes on the data. In unsupervised learning, of which clustering is the most important example, we have no such teacher to guide us.

The key input to a clustering algorithm is the distance measure. In Figure 16.1, the distance measure is distance in the two-dimensional (2D) plane.

Summary

Improving classifier effectiveness has been an area of intensive machine-learning research over the last two decades, and this work has led to a new generation of state-of-the-art classifiers, such as support vector machines, boosted decision trees, regularized logistic regression, neural networks, and random forests. Many of these methods, including support vector machines (SVMs), the main topic of this chapter, have been applied with success to information retrieval problems, particularly text classification. An SVM is a kind of large-margin classifier: It is a vector-space–based machine-learning method where the goal is to find a decision boundary between two classes that is maximally far from any point in the training data (possibly discounting some points as outliers or noise).

We will initially motivate and develop SVMs for the case of two-class data sets that are separable by a linear classifier (Section 15.1), and then extend the model in Section 15.2 to nonseparable data, multiclass problems, and nonlinear models, and also present some additional discussion of SVM performance. The chapter then moves to consider the practical deployment of text classifiers in Section 15.3: What sorts of classifiers are appropriate when, and how can you exploit domain-specific text features in classification? Finally, we will consider how the machine-learning technology that we have been building for text classification can be applied back to the problem of learning how to rank documents in ad hoc retrieval (Section 15.4).

Summary

Information retrieval (IR) systems are often contrasted with relational databases. Traditionally, IR systems have retrieved information from unstructured text – by which we mean “raw” text without markup. Databases are designed for querying relational data, sets of records that have values for predefined attributes such as employee number, title, and salary. There are fundamental differences between IR and database systems in terms of retrieval model, data structures, and query language as shown in Table 10.1.

Some highly structured text search problems are most efficiently handled by a relational database; for example, if the employee table contains an attribute for short textual job descriptions and you want to find all employees who are involved with invoicing. In this case, the SQL query:

select lastname from employees where job_desc like ‘invoic%’;

may be sufficient to satisfy your information need with high precision and recall.

STRUCTURED RETRIEVAL

However, many structured data sources containing text are best modeled as structured documents rather than relational data. We call the search over such structured documents structured retrieval. Queries in structured retrieval can be either structured or unstructured, but we assume in this chapter that the collection consists only of structured documents. Applications of structured retrieval include digital libraries, patent databases, blogs, text in which entities like persons and locations have been tagged (in a process called named entity tagging), and output from office suites like OpenOffice that save documents as marked up text.

Knowledge Management, Databases and Data Mining

Refine search

Refine search

Actions for selected content:

1911 results in Knowledge Management, Databases and Data Mining

Appendix: Additional Copyright Notices

Summary

4 - Query Specification

Summary

Preface

Summary

2 - The Evaluation of Search User Interfaces

Summary

6 - Query Reformulation

Summary

Contents

Author Index

The Text Mining Handbook

2 - The term vocabulary and postings lists

Summary

1 - Boolean retrieval

Summary

Table of Notation

18 - Matrix decompositions and latent semantic indexing

Summary

13 - Text classification and Naive Bayes

Summary

Contents

3 - Dictionaries and tolerant retrieval

Summary

20 - Web crawling and indexes

Summary

6 - Scoring, term weighting, and the vector space model

Summary

16 - Flat clustering

Summary

15 - Support vector machines and machine learning on documents

Summary

10 - XML retrieval

Summary

Knowledge Management, Databases and Data Mining

Refine search

Refine search

Actions for selected content:

Save Search

1911 results in Knowledge Management, Databases and Data Mining

Summary

Summary

Summary

Summary

Summary

The Text Mining Handbook

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary