Introduction

The jewel beetle Julodimorpha bakewelli is category challenged (Gwynne and Rentz 1983; Gwynne 2003). For the male of the species, spotting instances of the category “desirable female” is a pursuit of enduring interest and, to this end, he scours his environment for telltale signs of a female's shiny, dimpled, yellow-brown elytra (wing cases). Unfortunately for him, many males of the species Homo sapiens, who sojourn in his habitats within the Dongara area of Western Australia, are attracted by instances of the category “full beer bottle” but not by instances of the category “empty beer bottle,” and are therefore prone to toss their emptied bottles (stubbies) unceremoniously from their cars. As it happens, stubbies are shiny, dimpled, and just the right shade of brown to trigger, in the poor beetle, a category error. Male beetles find stubbies irresistible. Forsaking all normal females, they swarm the stubbies, genitalia everted, and doggedly try to copulate despite repeated glassy rebuffs. Compounding misfortune, ants of the species Iridomyrmex discors capitalize on the beetles' category errors; the ants sequester themselves near stubbies, wait for befuddled beetles, and consume them, genitalia first, as they persist in their amorous advances.

Categories have consequences. Conflating beetle and bottle led male J. bakewelli into mating mistakes that nudged their species to the brink of extinction.

The Problem of Object Discovery

It is perhaps not inaccurate to say that much of the brain's machinery is, in essence, devoted to the detection of repetitions in the environment. Knowing that a particular entity is the same as the one seen on a previous occasion allows the organism to put into play a response appropriate to the reward contingencies associated with that entity. This entity can take many forms. It can be a temporally extended event, such as an acoustic phrase or a dance move, a location, such as a living room, or an individual object, such as a peacock or a lamp. Irrespective of what the precise entity is, the basic informationprocessing problem is the same – to discover how the complex sensory input can be carved into distinct entities and to recognize them on subsequent occasions. In this chapter, we examine different pieces of knowledge regarding visual object discovery and attempt to synthesize them into a coherent framework.

Let us examine the challenges inherent in this problem a little more closely. Consider Figure 16.1(a). This complex landscape appears to be just a random collection of peaks and valleys, with no clearly defined groups. Yet it represents an image that is easily parsed into distinct objects by the human visual system. Figure 16.1(b) shows the image that corresponds exactly to the plot shown in Figure 16.1(a), which simply represents pixel luminance by height. It is now trivial for us to notice that there are three children, and we can accurately delimit their extents as shown in Figure 16.1(c).

Introduction

In Chapter 1, Dickinson analyzes the complex history of theoretical and computational vision. With some exceptions, the trend in recent decades is away from explicit structural representation and toward direct mapping of image features to semantic categories based on machine learning. The best-known formulations of the older, structural paradigm are those of Marr (Marr and Nishihara 1978) and Biederman (1987), although the central idea that objects are represented as configurations of parts has a long history (Barlow 1972; Binford 1971; Dickinson, Pentland, and Rosenfeld 1992; Hoffman and Richards 1984; Hubel and Wiesel 1959, 1968; Milner 1974; Palmer 1975; Selfridge 1959; Sutherland 1968). A configural representation would be carried by ensembles of processing units or neurons, each encoding the shape and relative position of a constituent part. This coding format is appealing because it solves three major problems in object vision. The first problem is the enormous dimensionality (on the order of 106) of retinal activity patterns. A signal of this complexity is too unwieldy to communicate between brain regions (owing to wiring constraints) or store in memory (owing to limited information capacity of synaptic weight patterns). Compression of this signal into a list of part specifications on the order of 101 to 102 would make communication and storage more practical. The second problem is the extremely variable mapping between retinal images and object identity. The same object can produce an infinity of very different retinal images depending on its position, orientation, lighting, partial occlusion, and other factors.

Introduction

Visual scenes are often complex and ambiguous to interpret because of the myriad causes that generate them. To understand visual scenes, our visual systems have to rely on our prior experience and assumptions about the world. These priors are rooted in the statistical correlation structures of visual events in our experience. They can be learned and exploited for probabilistic inference in a Bayesian framework using graphical models. Thus, we believe that understanding the statistics of natural scenes and developing graphical models with these priors for inference are crucial for gaining theoretical and computational insights to guide neurophysiological experiments. In this chapter, we will provide our perspective based on our work on scene statistics, graphical models, and neurophysiological experiments.

An important source of statistical priors for inference is the statistical correlation of visual events in our natural experience. In fact, it has long been suggested in the psychology community that learning due to coherent covariation of visual events is crucial for the development of Gestalt rules (Koffka 1935) as well as models of objects and object categories in the brain (Gibson 1979; Roger and McClelland 2004). Nevertheless, there has been relatively little research on how correlation structures in natural scenes are encoded by neurons. Here, we will first describe experimental results obtained from multielectrode neuronal recording in the primary visual cortex of awake-behaving monkeys. Each study was conducted on at least two animals. These results reveal mechanisms at the neuronal level for the encoding and influence of scene priors in visual processing.

Introduction

In our everyday interactions we encounter a plethora of novel experiences in different contexts that require prompt decisions for successful actions and social interactions. Despite the seeming ease with which we perform these interactions, extracting the key information from the highly complex input of the natural world and deciding how to classify and interpret it is a computationally demanding task for the primate visual system. Accumulating evidence suggests that the brain's solution to this problem relies on the combination of sensory information and previous knowledge about the environment. Although evolution and development have been suggested to shape the structure and organization of the visual system (Gilbert et al. 2001a; Simoncelli and Olshausen 2001), learning through everyday experiences has been proposed to play an important role in the adaptive optimization of visual functions. In particular, numerous behavioral studies have shown experience-dependent changes in visual recognition using stimuli ranging from simple features, such as oriented lines and gratings (Fahle 2004), to complex objects (Fine and Jacobs 2002). Recent neurophysiological (Logothetis et al. 1995; Rolls 1995; Kobatake et al. 1998; Rainer and Miller 2000; Jagadeesh et al. 2001; Schoups et al. 2001b; Baker et al. 2002; Ghose et al. 2002; Lee et al. 2002; Sigala and Logothetis 2002; Freedman et al. 2003; Miyashita 2004; Rainer et al. 2004; Yang and Maunsell 2004) and functional magnetic resonance imaging (fMRI) investigations (Dolan et al. 1997; Gauthier et al. 1999; Schiltz et al. 1999; Grill-Spector et al. 2000; van Turennout et al. 2000; Furmanski et al. 2004; Kourtzi et al. 2005b) have focused on elucidating the loci of brain plasticity and changes in neuronal responses that underlie this visual learning.

Introduction

In current classification schemes, the goal is usually to identify category instances in an image, together with their corresponding image locations. However, object recognition goes beyond top-level category labeling: when we see a known object, we not only recognize the complete object, but also identify and localize its parts and subparts at multiple levels. Identifying and localizing parts, called “object interpretation,” is often necessary for interacting with visible objects in the surrounding environment.

In this chapter I will describe a method for obtaining detailed interpretation of the entire object, by identifying and localizing parts at multiple levels. The approach has two main components. The first is the creation of a hierarchical feature representation that is constructed from informative parts and subparts, that are identified during a learning stage. The second is the detection and localization of objects and parts using a two-pass algorithm that is applied to the feature hierarchy. The resulting scheme has two main advantages. First, the overall recognition performance is improved compared with similar nonhierarchical schemes. Second, and more important, the scheme obtains reliable detection and localization of object parts even when the parts are locally ambiguous and cannot be recognized reliably on their own.

The second part of the chapter will discuss a possible future direction for improving the performance obtained by current classification methods, by the use of a continuous online model update, in an attempt to narrow the so-called performance gap between computational methods and human performance.

Introduction

The recognition of objects in images is a central task in computer vision. It is particularly challenging because the form and appearance of 3-D objects projected onto 2-D images undergo significant variation owing to the numerous dimensions of visual transformations, such as changes in viewing distance and direction, illumination conditions, occlusion, articulation, and, perhaps most significantly, within-category type variation. The challenge is how to encode the collection of these forms and appearances (Fig. 23.1), which are high-dimensional manifolds embedded in even higherdimensional spaces, such that the topology induced by such variations is preserved. We believe this is the key to the successful differentiation of various categories.

Form and appearance play separate, and distinct, but perhaps interacting roles in recognition. The early exploration of the form-only role assumed that figures can be successfully segregated from image. This assumption was justified by the vast platform of “segmentation” research going on at the same time. Currently, it is generally accepted that segmentation as a stand-alone approach is ill-posed and that segmentation must be approached together with recognition or other high-level tasks. Nevertheless, research on form-only representation and recognition remains immensely valuable in that it has identified key issues in shape representation and key challenges in recognition such as those in the presence of occlusion, articulation, and other unwieldy visual transformations, which are present even in such a highly oversimplified domain. Section 23.2 reviews some of the lessons learned and captured in the context of the shock-graph approach to recognition.

Introduction

Allow me to pose a few questions on the nature of visual recognition. Why do we bother studying it? What are the “things” we recognize? How many things is it useful to recognize? What is the nature of different recognition tasks? What have we learned so far? What are the open problems that face us? I am assuming that the reader has some familiarity with the technical aspects of vision and visual recognition. This is not a survey, and the references are meant to exemplify an idea or an approach; they are not meant to give proper credit to the many excellent people who work in the field. Also, some of the interesting technical issues are not visible from the mile-high perspective I take here and are therefore not mentioned.

What?

What is it that we recognize in images? We recognize both the component elements and the overall scene: materials and surface properties (“leather,” “wet”), objects (“frog,” “corkscrew”) and the gist of the ensemble (“kitchen,” “prairie”). We recognize things both as individuals (“Gandhi,” “my bedroom”) and as members of categories (“people,” “mountainscape”). These distinctions are important because the visual statistics of materials are different from those of objects and scenes. Also, the visual variability of individual objects is different from that of categories; different approaches may be needed to model and recognize each. In the following text, for brevity, when referring to the thing to be recognized, I will often call it “object,” although it could be a material or a scene, an individual or a category.

Introduction

Visual classification presents many challenges, but in this chapter we will focus on the problem of generalization. This is the ability to learn about a new class of object from some images, and then to recognize instances of that class in very different images. To do this we must somehow account for variations in appearance due to changes in viewpoint, alterations in lighting, variations in the shape of different objects within the same class, and a myriad of other causes.

At the core of this generalization is the question: how best can we determine the similarity between a small set of 2-D images? Most of our visual input comes in the form of 2-D images. Our ability to classify is based on experience with a large number of images, but if we want to know how to get the most from many images, it makes sense to begin by asking how we can get the most from just a few images.

One of the most important constraints that we can use when comparing 2-D images is our knowledge that these are images of the 3-D world. We do not need to treat images as generic 2-D signals, and to compare them as such. Rather, we can base our comparisons on knowledge derived from the 3-D world. Geometry provides strong constraints on how the 2-D appearance of a 3-D object can change with viewpoint. The physics of light and the material properties of objects that reflect this light constrain how much variation there can be in the appearance of an object as the lighting varies.

Introduction

Object representations for categorization tasks should be applicable for a wide range of objects, scalable to handle large numbers of object classes, and at the same time learnable from a few training samples. While such a scalable representation is still illusive today, it has been argued that such a representation should have at least the following properties: it should enable sharing of features (Torralba et al. 2007), it should combine generative models with discriminative models (Fritz et al. 2005; Jaakkola and Haussler 1999), and it should combine both local and global as well as appearanceand shape-based features (Leibe et al. 2005). Additionally, we argue that such object representations should be applicable both for unsupervised learning (e.g., visual object discovery) as well as supervised training (e.g., object detection). Therefore, we extend our previous efforts of hybrid modeling (Fritz et al. 2005) with ideas of unsupervised learning of generative decompositions to obtain an approach that integrates across different paradigms of modeling, representing, and learning of visual categories.

We present a novel method for the discovery and detection of visual object categories based on decompositions using topic models. The approach is capable of learning a compact and low-dimensional representation for multiple visual categories from multiple viewpoints without labeling of training instances. The learnt object components range from local structures over line segments to global silhouette-like descriptions. This representation can be used to discover object categories in a totally unsupervised fashion. Furthermore we employ the representation as the basis for building a supervised multicategory detection system making efficient use of training examples and outperforming pure features-based representations.

Introduction

Understanding how the brain performs object categorization is of significant interest for cognitive neuroscience as well as for machine vision. In the past decade, building on earlier efforts (Fukushima 1980; Perrett and Oram 1993; Wallis and Rolls 1997), there has been significant progress in understanding the neural mechanisms underlying object recognition in the brain (Kourtzi and DiCarlo 2006; Peissig and Tarr 2007; Riesenhuber and Poggio 1999b; Riesenhuber and Poggio 2000; Riesenhuber and Poggio 2002; Serre et al. 2007a). There is now a quantitative computational model of the ventral visual pathway in primates that models rapid object recognition (Riesenhuber and Poggio 1999b; Serre et al. 2007a), putatively based on a single feedforward pass through the visual system (Thorpe and Fabre-Thorpe 2001). The model has been validated through a number of experiments that have confirmed nontrivial qualitative and quantitative predictions of the model (Freedman et al. 2003; Gawne and Martin 2002; Jiang et al. 2007; Jiang et al. 2006; Lampl et al. 2004). In the domain of machine vision, the success of the biological model in accounting for human object recognition performance (see, e.g., (Serre et al. 2007b)) has led to the development of a family of biologically inspired machine vision systems (see, e.g., (Marin-Jimenez and Perez de la Blanca, 2006; Meyers and Wolf, 2008; Mutch and Lowe, 2006; Serre et al. 2007c)).

Building upon the fast growing technological advance of video compression in the 1980s, along with the availability of affordable fast computing processors and digital memories in the early 1990s, the evolution in use of digital multimedia broadcasting proceeded rapidly (see Table 6.1). The arrival of digital broadcasting was significant; what was happening was not just a simple move from an analog system to a digital system. Rather, digital broadcasting permits a level of quality and flexibility unattainable with analog broadcasting and provides a wide range of convenient services, thanks to its high picture and sound quality, interactivity, and storage capability. European broadcasters initiated the first attempt to implement a complete direct-to-home satellite digital television program delivery infrastructure having a capacity in excess of 100 channels from a single satellite. This was the digital video broadcasting (DVB) project in 1993, and the main standardization work for satellite (DVB-S) and cable (DVB-C) delivery systems was completed in 1994 [1] [2]. The fixed terrestrial version (DVB-T) was soon added to the DVB family to offer one-to-many broadband wireless data broadcasting based on roof-top antenna and the use of IP packets.

All these DVB sub-standards basically differ only in the specifications to the physical representation, modulation, transmission, and reception of the signal. Digital video broadcasting is, however, much more than a simple replacement for existing analog television transmission. More specifically, DVB provides superior picture quality with the opportunity to view pictures in standard format or wide screen (16:9) format, along with mono, stereo, or surround sound.

The rapid growth of wireless broadband networking infrastructures, such as 3G and 3.5G, WLAN and WLAN-mesh, and WiMAX, makes available multimedia (audio and video) information and entertainment (“infotainment”) in our lives anytime, anywhere, on any device. However, wireless multimedia delivery faces several challenges, such as a high error rate, bandwidth variation and limitation, battery power limitation, and so on. Take, for example, the voice over IP (VoIP) and video streaming applications, which are quite mature in wireline infrastructure. At the same time, wireless broadband based on WLAN and WiMAX is also becoming widespread. While these wireless networks were not designed with real-time multimedia communication services in mind, their widespread availability and low cost makes them an inviting solution for adding mobility to these communication services. The major issue is how to achieve a wireless broadband system which can deliver real-time interactive multimedia smoothly and still satisfy the QoS metrics typically used to define the quality of a VoIP or video conferencing session, e.g., the one-way delay, jitter, packet loss rate, and throughput (see Section 7.2).

Advances in media coding over wireless networks are governed by two dominant rules [1]. One is the well-known Moore's law, which states that computing power doubles every 18 months. Moore's law certainly applies to media codec evolution, and there have been huge advances in technology in the ten years since the adoption of MPEG-2. The second governing principle is the huge bandwidth gap (one or two orders of magnitude) between wireless and wired networks. This bandwidth gap demands that coding technologies must achieve efficient compact representation of media data over wireless networks.

Owing to the proliferation of digitized media applications, such as e-Book, streaming videos, web images, shared music, etc., there is a growing need to protect the intellectual property rights of digital media and prevent illegal copying and falsification. This explains the strong demand for digital rights management (DRM), which is an access control technology that protects and enforces the rights associated with the use of digital content, such as multimedia data. The most important functions of DRM are to prevent unauthorized access and the creation of unauthorized copies of digital content, and moreover to provide a mechanism by which copies can be detected and traced (content tracking). Digital rights management is the most critical component of the intellectual property management and protection (IPMP) protocol widely promoted in the MPEG standards. Under the IPMP's scope, intellectual property (IP) is anything whose use owes the inventors or the owners some form of compensation. This could be a particular media application or it could be the technology used by the media. The management of IP involves storage and serving, appropriate authorization of use, and correct billing and tracking. The protection of IP prevents unauthorized use or misuse of the IP and can make legitimate use easy. According to [1], an effective DRM system should have the following four requirements.

1. (1) The DRM system must package the content to be protected in a secure manner.

2. (2) The DRM system must obtain the access conditions (license) specified by the owner of the protected content.

3. (3) The DRM system must determine whether the access conditions have been fulfilled

4. […]

With the great advances in digital data compression (coding) technologies and the rapid growth in the use of IP-based Internet, along with the quick deployment of last-mile wire line and wireless broadband access, networked multimedia applications have created a tremendous impact on computing and network infrastructures. The four most critical and indispensable components involved in a multimedia networking system are: (1) data compression (source encoding) of multimedia data sources, e.g., speech, audio, image, and video; (2) quality of service (QoS) streaming architecture design issues for multimedia delivery over best-effort IP networks; (3) effective dissemination multimedia over heterogeneous IP wireless broadband networks, where the QoS is further degraded owing to the dynamic changes in end-to-end available bandwidth caused by wireless fading or shadowing and link adaptation; (4) effective digital rights management and adaptation schemes, which are needed to ensure proper intellectual property management and protection of networked multimedia content.

This book has been written to provide an in-depth understanding of these four major considerations and their critical roles in multimedia networking. More specifically, it is the first book to provide a complete system design perspective based on existing international standards and state-of-the-art networking and infrastructure technologies, from theoretical analyses to practical design considerations. The book also provides readers with learning experiences in multimedia networking by offering many development-software samples for multimedia data capturing, compression, and streaming for PC devices, as well as GUI designs for multimedia applications. The coverage of the material in this book makes it appropriate as a textbook for a one-semester or two-quarter graduate course.