In this chapter, we define an abstraction of the sentiment analysis problem. This abstraction gives us a statement of the problem and enables us to see a rich set of interrelated subproblems. It is often said that if we cannot structure a problem, we probably do not understand the problem. The objective of the definitions is thus to abstract a structure from the complex and intimidating unstructured natural language text. The structure serves as a common framework to unify various existing research directions and enable researchers to design more robust and accurate solution techniques by exploiting the interrelationships of the subproblems. From a practical application point of view, the definitions let practitioners see what subproblems need to be solved in building a sentiment analysis system, how the subproblems are related, and what output should be produced.

Unlike factual information, sentiment and opinion have an important characteristic, namely, they are subjective. The subjectivity comes from many sources. First of all, different people may have different experiences and thus different opinions. For example, one person bought a camera of a particular brand and had a very good experience with it. She naturally has a positive opinion or sentiment about the camera. However, another person who also bought a camera of the same brand had some issues with it because he might just be unlucky and got a defective unit. He thus has a negative opinion. Second, different people may see the same thing in different ways because everything has two sides. For example, when the price of a stock is falling, one person may feel very sad because he bought the stock when the price was high, but another person may be very happy because it is an opportunity to short sell the stock to make good profits. Furthermore, different people may have different interests and/or different ideologies.

Before performing an action, we almost always have the intention to perform the action first. In many cases, we also talk about or write about our intentions. Although the concept of intention has been investigated in philosophy and psychology, researchers in these fields are usually not concerned with the language used to express intentions or how to infer intentions from written language computationally, which is our objective in this chapter. Studying intention computationally is just beginning, and our understanding of the problem is still limited.

Intention and sentiment are generally regarded as two different concepts. They are, however, closely related, which we will discuss in Section 11.1. Mining intentions also has many practical applications. For example, if we find that a large number of Twitter posts say something like “I am dying to see the Life of Pi,” we can predict that the movie is going to do well at the box office. If one wrote “I am looking for a car to replace my old Ford Focus, any suggestions?” the person clearly wants to buy a car and a car dealer can quickly recommend the person some new car models. If the dealer also has reviews of these cars, these reviews can be shown to this potential buyer in persuasion at the same time. Mining intention information is also useful to social media hosting sites, which often use their user-generated content (posts) for advertising. If user intentions can be recognized automatically, advertising can be made intention based and should therefore be much more targeted and effective. This chapter is organized as follows. Section 11.1 defines the intention mining problem. Section 11.2 studies an existing method for identifying intention posts in social media based on transfer learning. Section 11.3 suggests a simple method to mine fine-grained intentions of any kind in a massive scale from any social media site.

This book introduced the field of sentiment analysis or opinion mining. It presented some basic knowledge and mature techniques in detail and surveyed numerous other state-of-the-art algorithms. Owing to numerous challenging research problems and a wide variety of practical applications, sentiment analysis has been a very active research area in several computer science fields, for example, NLP, data mining, web mining, and information retrieval. It has also spread to management science (Hu et al., 2006; Archak et al., 2007; Das and Chen, 2007; Dellarocas et al., 2007; Ghose et al., 2007; Park et al., 2007; Chen and Xie, 2008) and other social science fields such as communications and political science because of its importance to business and society as a whole. With the rapid expansion of social media on the web, the importance of sentiment analysis is also growing by the day.

In the book, I first defined the problem of sentiment analysis (Chapter 2), which provides a common framework to unify different research directions and research problems in the area. It also presents a schema to convert unstructured free text to structured data, which facilitates qualitative and quantitative analysis of opinions. I then discussed the widely studied problem of document-level sentiment classification (Chapter 3), which aims to determine whether an opinion document (e.g., a product review) expresses a positive or a negative sentiment. This was followed by Chapter 4 on sentence-level subjectivity and sentiment classification. These tasks determine whether a sentence is opinionated, and if opinionated, whether it carries a positive or negative opinion. Chapters 5 and 6 focused on aspect-based sentiment analysis, which employs the full problem definition in Chapter 2. They also showed that sentiment analysis is a multifaceted problem with multiple challenging subproblems. The existing techniques for dealing with these problems were also presented. Due to a large amount of information, this topic was covered in two chapters. Chapter 5 was dedicated to the task of aspect-based sentiment classification, and Chapter 6 was devoted to the task of opinion target extraction, which includes the extraction of entities and their aspects.

In this chapter, we discuss the quality of reviews. The topic is related to opinion spam detection, but is also very different because low-quality reviews may not be spam or fake reviews, and fake reviews may not be perceived as low-quality reviews by readers because as we discussed in the last chapter, it is very difficult to spot fake reviews simply by reading them. For this reason, fake reviews may also be seen as helpful or high-quality reviews if the imposters write their reviews early and craft them well.

The objective of this task is to determine the quality, helpfulness, usefulness, or utility of each review (Kim et al., 2006; Zhang and Varadarajan, 2006; Ghose and Ipeirotis, 2007; Liu et al., 2007). This is a meaningful task because it is desirable to rank reviews based on quality or helpfulness when showing reviews to the user, with the most helpful reviews first. In fact, many review aggregation or hosting sites have been practicing this for years. They obtain the helpfulness or quality score of each review by asking readers to provide helpfulness feedback to each review. For example, in Amazon.com, the reader can indicate whether she finds a review helpful by responding to the question “Was the review helpful to you?” just below each review. The feedback results from all responses are then aggregated and displayed right before each review, for example, “15 of 16 people found the following review helpful.” Although most review hosting sites already provide the service, automatically determining the quality of each review is still useful because a good number of user feedback may take a long time to accumulate. That is why many reviews have few or no feedback. This is especially true for new reviews.

Quality Prediction as a Regression Problem

Determining the quality of reviews is usually formulated as a regression problem. The learned model assigns a quality score to each review, which can be used in Review ranking or review recommendation.

Opinion and sentiment and their related concepts, such as evaluation, appraisal, attitude, affect, emotion, and mood, are about our subjective feelings and beliefs. They are central to human psychology and are key influencers of our behaviors. Our beliefs and perceptions of reality, as well as the choices we make, are to a considerable degree conditioned on how others see and perceive the world. For this reason, our views of the world are very much influenced by others' views, and whenever we need to make a decision, we often seek out others' opinions. This is true not only for individuals but also for organizations. From an application point of view, we naturally want to mine people's opinions and feelings toward any subject matter of interest, which is the task of sentiment analysis. More precisely, sentiment analysis, which is also called opinion mining, is a field of study that aims to extract opinions and sentiments from natural language text using computational methods.

The inception and rapid growth of sentiment analysis coincide with those of social media on the web, such as reviews, forum discussions, blogs, and microblogs, because for the first time in human history, we now have a huge volume of opinion data recorded in digital forms. These data, also called user-generated content, prompted researchers to mine them to discover useful knowledge. This naturally led to the problem of sentiment analysis or opinion mining because these data are full of opinions. That these data are full of opinions is not surprising, because the primary reason why people post messages on social media platforms is to express their views and opinions, and therefore sentiment analysis is at the very core of social media analysis. Since early 2000, sentiment analysis has grown to be one of the most active research areas in natural language processing. It is also widely studied in data mining, web mining, and information retrieval. In fact, the research has spread from computer science to management science and social science because of its importance to business and society as a whole. In recent years, industrial activities surrounding sentiment analysis have also thrived. Numerous start-ups have emerged.

Apart from directly expressing positive or negative opinions about an entity and/or its aspects, one can also express opinions by comparing similar entities. Such opinions are called comparative opinions (Jindal and Liu, 2006a, 2006b). Comparative opinions have different semantic meanings than regular opinions and also different syntactic forms. For example, a typical regular opinion sentence is “The voice quality of this phone is amazing,” and a typical comparative opinion sentence is “The voice quality of Moto X is better than that of iPhone 5.” This comparative sentence does not say that any phone's voice quality is good or bad, but simply states a relative ordering in terms of voice quality of the two smart phones. Like regular sentences, comparative sentences can be opinionated or not-opinionated. The above comparative sentence is clearly opinionated because it explicitly expresses a comparative sentiment, while the sentence “Samsung Galaxy 4 is larger than iPhone 5” expresses no sentiment, at least not explicitly. In this chapter, we first define the problem of comparative opinion mining and then present some existing methods for solving the problem. We will study superlative opinions as well because their semantic meanings and handling methods are similar.

Problem Definition

A comparative sentence usually expresses a relation based on the similarities or differences of more than one entity. Linguists have studied comparatives in the English language for a long time. Lerner and Pinkal (1992) defined comparatives as universal quantifiers over degrees. For example, in the sentence “John is taller than he was,” the degree d is John's height and John is tall to degree d. In other words, comparatives are used to express explicit orderings between objects with respect to the degree or amount to which they possess some gradable property (Kennedy, 2005). Two broad types of comparatives are as follows:

1. Metalinguistic comparatives. Compare the extent to which an entity has one property to a greater or lesser extent than another property, for example, “Ronaldo is angrier than upset.”

2. […]

Opinions from social media are increasingly used by individuals and organizations for making purchase decisions, making choices at elections, and for marketing and product design. Positive opinions often mean profits and fame for businesses and individuals. This, unfortunately, gives strong incentives for imposters to game the system by posting fake reviews or opinions to promote or to discredit some target products, services, organizations, individuals, and even ideas without disclosing their true intentions, or the person or organization that they are secretly working for. Such individuals are called opinion spammers and their activities are called opinion spamming (Jindal and Liu, 2007, 2008). An opinion spammer is also called a shill, a plant, or a stooge in the social media environment, and opinion spamming is also called shilling or astroturfing. Opinion spamming not only can hurt consumers and damage businesses, but also can warp opinions and mobilize masses into positions counter to legal or ethical mores. This can be frightening, especially when spamming is about opinions on social and political issues. It is safe to say that as opinions in social media are increasingly used in practice, opinion spamming is becoming more and more sophisticated, which presents a major challenge for their detection. However, they must be detected to ensure that the social media continues to be a trusted source of public opinions, rather than being full of fakes, lies, and deceptions. The good news is that both the industry and the research community have made tremendous progress in combating opinion spamming. I am aware that several major review hosting sites are able to detect a good proportion of fake reviews and fake reviewers. These efforts have already acted as a deterrent to opinion spamming and made it difficult for inexperienced spammers to succeed. However, the problem is still huge and a great deal of research is needed.

Spam detection in general has been studied in many fields. Web spam and e-mail spam are perhaps the two most widely studied types of spam. Opinion spam is, however, very different.

Sentiment analysis, also called opinion mining, is the field of study that analyzes people's opinions, sentiments, appraisals, attitudes, and emotions toward entities and their attributes expressed in written text. The entities can be products, services, organizations, individuals, events, issues, or topics. The field represents a large problem space. Many related names and slightly different tasks, for example, sentiment analysis, opinion mining, opinion analysis, opinion extraction, sentiment mining, subjectivity analysis, affect analysis, emotion analysis, and review mining, are now all under the umbrella of sentiment analysis. The term sentiment analysis perhaps first appeared in Nasukawa and Yi (2003), and the term opinion mining first appeared in Dave et al. (2003). However, research on sentiment and opinion began earlier (Wiebe, 2000; Das and Chen, 2001; Tong, 2001; Morinaga et al., 2002; Pang et al., 2002; Turney, 2002). Even earlier related work includes interpretation of metaphors; extraction of sentiment adjectives; affective computing; and analysis of subjectivity, viewpoints, and affects (Wiebe, 1990, 1994; Hearst, 1992; Hatzivassiloglou and McKeown, 1997; Picard, 1997; Wiebe et al., 1999). An early patent on text classification included sentiment, appropriateness, humor, and many other concepts as possible class labels (Elkan, 2001). Since existing research and applications of sentiment analysis have focused primarily on written text, it has been an active research field of natural language processing (NLP). However, the topic has also been widely studied in data mining, web mining, and information retrieval because many researchers in these fields deal with text data. My own first paper (Hu and Liu, 2004) on the topic was published in the proceedings of the data mining conference KDD (SIGKDD International Conference on Knowledge Discovery and Data Mining) in 2004. This paper defined the aspect-based sentiment analysis and summarization framework and some basic ideas and algorithms for solving the problem that are commonly used in research and industrial systems today.

Not surprisingly, there has been some confusion among practitioners and even researchers about the difference between sentiment and opinion and whether the field should be called sentiment analysis or opinion mining.

This is the second chapter about aspect-based sentiment analysis. In Section 2.1, we defined each opinion as a quintuple (e, a, s, h, t), where e is an entity and a is one of its aspects, s is the sentiment on the aspect a, h is the opinion holder and t is the time when the opinion is expressed. Chapter 5 focused on aspect-based sentiment classification, which determines s. This chapter focuses on extraction of entities and aspects on which sentiments or opinions have been expressed.

Entity and aspect extractions are often regarded as two separate tasks because the methods and features used for their recognition are usually different due to their individual specific characteristics. Entities commonly refer to names of products, services, individuals, events, and organizations, and aspects commonly refer to the attributes and components of entities. The two tasks are also collectively called opinion target extraction in sentiment analysis because they together form the targets of opinions. After their extraction, a resolution step is also performed to group synonymous entities and aspects together to facilitate opinion summarization. Let us use two example sentences to ground the tasks:

The entities in the two sentences are Motorola X and Moto X, and the aspects are voice quality and sound. The entity and aspect extraction tasks aim to find these entities and aspects. The resolution step should group Motorola X and Moto X together as they refer to the same entity, and voice quality and sound together as they refer to the same aspect. This chapter will not discuss how to determine sentiments on entities or aspects as it has been studied in Chapter 5.

In general, both aspect extraction and entity extraction are information extraction tasks. However, in the context of sentiment analysis, some specific characteristics of the problem can facilitate their extractions. A key characteristic is that an opinion always has a target. The target is an aspect or an entity.

Starting from this chapter, we discuss the main research topics of sentiment analysis and their state-of-the-art algorithms. Document sentiment classification (or document-level sentiment analysis) is perhaps the most extensively studied topic in the field of sentiment analysis especially in its early days (see surveys by Pang and Lee, 2008; Liu, 2012). It aims to classify an opinion document (e.g., a product review) as expressing a positive or a negative opinion (or sentiment), which are called sentiment orientations or polarities. The task is referred to as document-level analysis because it considers each document as a whole and does not study entities or aspects inside the document or determine sentiments expressed about them. Arguably, this is the task that popularized sentiment analysis research. Its limitations also motivated the fine-grained task of aspect-based sentiment analysis (Hu and Liu, 2004) (Chapters 5 and 6) that is widely used in practice today.

Document sentiment classification is considered the simplest sentiment analysis task because it treats sentiment classification as a traditional text classification problem with sentiment orientations or polarities as the classes. Thus, any supervised learning algorithms can be applied directly to solve the problem. In most cases, the features used in classification are the same as those used in traditional text classification too. Owing to its simple problem definition and equivalence to text classification, it has served as the base task of several other research directions adapted from the general text classification, for example, cross-lingual and cross-domain sentiment classification.

To ensure that the task is meaningful in practice, existing literature on document sentiment classification makes the following implicit assumption (Liu, 2010).

Assumption 3.1: Document sentiment classification assumes that the opinion document d (e.g., a product review) expresses opinions on a single entity e and contains opinions from a single opinion holder h.

Thus, strictly speaking, document sentiment classification can only be applied to a special type of opinion documents. We make this assumption explicit in the task definition.

By now, it should be quite clear that words and phrases that convey positive or negative sentiment are instrumental for sentiment analysis. This chapter discusses how to compile such word lists. In the research literature, sentiment words are also called opinion words, polar words, or opinion-bearing words. Positive sentiment words such as beautiful, wonderful, and amazing are used to express some desired states or qualities while negative sentiment words such as bad, awful, and poor are used to express some undesired states or qualities. In addition to individual words, there are also sentiment phrases and idioms, for example, cost an arm and a leg. Collectively, they are called the sentiment lexicon (or opinion lexicon). From now on, when we say sentiment words, we mean both individual words and phrases.

Sentiment words can be divided into two types, base type and comparative type. All the preceding example words are of the base type. Sentiment words of the comparative type (which include the superlative type) are used to express comparative and superlative opinions. Examples of such words include better, worse, best, worst, and so on, which are comparative and superlative forms of the base adjectives or adverbs such as good and bad. We discuss comparative and superlative sentiment words further in Chapter 8. This chapter focuses on sentiment words of the base type.

There are three main existing approaches to compiling sentiment words: manual approach, dictionary-based approach (discussed in Section 7.1), and corpusbased approach (discussed in Section 7.2). The manual approach is labor-intensive and time-consuming, and is thus usually used as a check on automated approaches because automated approaches make mistakes. In Section 7.3, we will also discuss the issue of factual statements implying opinions, which has largely been overlooked by the research community.

This chapter is written in a survey style because past research has constructed numerous sentiment lexicons in many languages. Most of them are publicly available (see Section 7.4 for a list of them in English).

A fundamental data-mining problem is to examine data for “similar” items. We shall take up applications in Section 3.1, but an example would be looking at a collection of Web pages and finding near-duplicate pages. These pages could be plagiarisms, for example, or they could be mirrors that have almost the same content but differ in information about the host and about other mirrors.

We begin by phrasing the problem of similarity as one of finding sets with a relatively large intersection. We show how the problem of finding textually similar documents can be turned into such a set problem by the technique known as “shingling.” Then, we introduce a technique called “minhashing,” which compresses large sets in such a way that we can still deduce the similarity of the underlying sets from their compressed versions. Other techniques that work when the required degree of similarity is very high are covered in Section 3.9.

Another important problem that arises when we search for similar items of any kind is that there may be far too many pairs of items to test each pair for their degree of similarity, even if computing the similarity of any one pair can be made very easy. That concern motivates a technique called “locality-sensitive hashing,” for focusing our search on pairs that are most likely to be similar.

Finally, we explore notions of “similarity” that are not expressible as intersection of sets. This study leads us to consider the theory of distance measures in arbitrary spaces. It also motivates a general framework for locality-sensitive hashing that applies for other definitions of “similarity.”

Applications of Near-Neighbor Search

We shall focus initially on a particular notion of “similarity”: the similarity of sets by looking at the relative size of their intersection. This notion of similarity is called “Jaccard similarity,” and will be introduced in Section 3.1.1. We then examine some of the uses of finding similar sets.

Clustering is the process of examining a collection of “points,” and grouping the points into “clusters” according to some distance measure. The goal is that points in the same cluster have a small distance from one another, while points in different clusters are at a large distance from one another. A suggestion of what clusters might look like was seen in Fig. 1.1. However, there the intent was that there were three clusters around three different road intersections, but two of the clusters blended into one another because they were not sufficiently separated.

Our goal in this chapter is to offer methods for discovering clusters in data. We are particularly interested in situations where the data is very large, and/or where the space either is high-dimensional, or the space is not Euclidean at all. We shall therefore discuss several algorithms that assume the data does not fit in main memory. However, we begin with the basics: the two general approaches to clustering and the methods for dealing with clusters in a non-Euclidean space.

Introduction to Clustering Techniques

We begin by reviewing the notions of distance measures and spaces. The two major approaches to clustering – hierarchical and point-assignment – are defined. We then turn to a discussion of the “curse of dimensionality,” which makes clustering in high-dimensional spaces difficult, but also, as we shall see, enables some simplifications if used correctly in a clustering algorithm.

7.1.1 Points, Spaces, and Distances

A dataset suitable for clustering is a collection of points, which are objects belonging to some space. In its most general sense, a space is just a universal set of points, from which the points in the dataset are drawn. However, we should be mindful of the common case of a Euclidean space (see Section 3.5.2), which has a number of important properties useful for clustering. In particular, a Euclidean space's points are vectors of real numbers.

Modern data-mining applications, often called “big-data” analysis, require us to manage immense amounts of data quickly. In many of these applications, the data is extremely regular, and there is ample opportunity to exploit parallelism. Important examples are:

1. (1) The ranking of Web pages by importance, which involves an iterated matrix-vector multiplication where the dimension is many billions.

2. (2) Searches in “friends” networks at social-networking sites, which involve graphs with hundreds of millions of nodes and many billions of edges.

To deal with applications such as these, a new software stack has evolved. These programming systems are designed to get their parallelism not from a “super-computer,” but from “computing clusters” – large collections of commodity hardware, including conventional processors (“compute nodes”) connected by Ethernet cables or inexpensive switches. The software stack begins with a new form of file system, called a “distributed file system,” which features much larger units than the disk blocks in a conventional operating system. Distributed file systems also provide replication of data or redundancy to protect against the frequent media failures that occur when data is distributed over thousands of low-cost compute nodes.

On top of these file systems, many different higher-level programming systems have been developed. Central to the new software stack is a programming system called MapReduce. Implementations of MapReduce enable many of the most common calculations on large-scale data to be performed on computing clusters efficiently and in a way that is tolerant of hardware failures during the computation.

MapReduce systems are evolving and extending rapidly. Today, it is common for MapReduce programs to be created from still higher-level programming systems, often an implementation of SQL. Further, MapReduce turns out to be a useful, but simple, case of more general and powerful ideas. We include in this chapter a discussion of generalizations of MapReduce, first to systems that support acyclic workflows and then to systems that implement recursive algorithms.

Our last topic for this chapter is the design of good MapReduce algorithms, a subject that often differs significantly from the matter of designing good parallel algorithms to be run on a supercomputer.

There are many sources of data that can be viewed as a large matrix. We saw in Chapter 5 how the Web can be represented as a transition matrix. In Chapter 9, the utility matrix was a point of focus. And in Chapter 10 we examined matrices that represent social networks. In many of these matrix applications, the matrix can be summarized by finding “narrower” matrices that in some sense are close to the original. These narrow matrices have only a small number of rows or a small number of columns, and therefore can be used much more efficiently than can the original large matrix. The process of finding these narrow matrices is called dimensionality reduction.

We saw a preliminary example of dimensionality reduction in Section 9.4. There, we discussed UV-decomposition of a matrix and gave a simple algorithm for finding this decomposition. Recall that a large matrix M was decomposed into two matrices U and V whose product UV was approximately M. The matrix U had a small number of columns whereas V had a small number of rows, so each was significantly smaller than M, and yet together they represented most of the information in M that was useful in predicting ratings of items by individuals.

In this chapter we shall explore the idea of dimensionality reduction in more detail. We begin with a discussion of eigenvalues and their use in “principal component analysis” (PCA). We cover singular-value decomposition, a more powerful version of UV-decomposition. Finally, because we are always interested in the largest data sizes we can handle, we look at another form of decomposition, called CUR-decomposition, which is a variant of singular-value decomposition that keeps the matrices of the decomposition sparse if the original matrix is sparse.

Eigenvalues and Eigenvectors

We shall assume that you are familiar with the basics of matrix algebra: multiplication, transpose, determinants, and solving linear equations for example. In this section, we shall define eigenvalues and eigenvectors of a symmetric matrix and show how to find them.

One of the big surprises of the 21st century has been the ability of all sorts of interesting Web applications to support themselves through advertising, rather than subscription. While radio and television have managed to use advertising as their primary revenue source, most media – newspapers and magazines, for example – have had to use a hybrid approach, combining revenue from advertising and subscriptions.

By far the most lucrative venue for on-line advertising has been search, and much of the effectiveness of search advertising comes from the “adwords” model of matching search queries to advertisements. We shall therefore devote much of this chapter to algorithms for optimizing the way this assignment is done. The algorithms used are of an unusual type; they are greedy and they are “on-line” in a particular technical sense to be discussed. We shall therefore digress to discuss these two algorithmic issues – greediness and on-line algorithms – in general, before tackling the adwords problem.

A second interesting on-line advertising problem involves selecting items to advertise at an on-line store. This problem involves “collaborative filtering,” where we try to find customers with similar behavior in order to suggest they buy things that similar customers have bought. This subject will be treated in Section 9.3.

Issues in On-Line Advertising

In this section, we summarize the technical problems that are presented by the opportunities for on-line advertising. We begin by surveying the types of ads found on the Web.

8.1.1 Advertising Opportunities

The Web offers many ways for an advertiser to show their ads to potential customers. Here are the principal venues.

(1) Some sites, such as eBay, Craig's List or auto trading sites allow advertisers to post their ads directly, either for free, for a fee, or a commission.

(2) Display ads are placed on many Web sites. Advertisers pay for the display at a fixed rate per impression (one display of the ad with the download of the page by some user). Normally, a second download of the page, even by the same user, will result in the display of a different ad and is a second impression.