This chapter provides an insight to some of the generalimage transforms that offer an alternativerepresentation of images and videos. Few of theirproperties and applications are also discussed thatare related to image compression and reconstruction.Other forms of representation that depend on data,like principal component analysis and sparserepresentation, are provided as an extension tothese representations. Techniques of computing basisfunctions and dictionary learning are introduced inthis chapter.

2.1 Image transforms

Consider a continuous function, 𝑓(𝑥), inone-dimensional (1-D) space, where, 𝑥 ∈ ℝ. Considera set, B, of 1-D basis functions, whose functionalvalues may either be in real or in complex domain.This is represented as in Eq. 2.1.

Clustering is a task oforganizing objects into groups whose members aresimilar in some way. A cluster is a collection of objects thatare similar to each other, but dissimilar to theobjects belonging to other clusters. In other words,a cluster is a group of objects with loosely definedsimilarity among them, which may have the potentialto form a class. A class is a known group of objects thatare described by similar characteristics, andclassification isthe task of assigning a defined class to an object.Image segmentationis also a problem that is similar to clustering,where the clusters are formed by groups of pixelsthat are similar in some context. In imagesegmentation, homogeneous regions in an image may beclustered to derive segments in the image. Thesesegments represent clusters. An example of imagesegmentation is shown in Fig. 6.1, where theforeground is represented by mushroom and thebackground is represented by humus substance aroundit. In this case, the image is primarily clusteredinto two regions, which are shown by white solidcontour (foreground) and white dashed contour(background).

The main motivations of clustering techniques are asfollows.

• • To find representative samples of homogeneousgroups in the given data, which would reduce thedata transmission and storage requirements incertain applications. Here, the data isrepresented by a smaller set of representativesamples that capture the characteristics of totaldata.

• • To discover natural groups or categories inthe data, which may be used to describe the datasamples by their unknown properties.

• • To find relevant groups in the data, whichfacilitates to draw attention toward major groupsof the data in the distribution. These groups formthe major clusters in a given context, likesegments in an image.

• • To detect unusual data objects, which are theoutliers in the data, that deviate from thecollective characteristics of groups of data in agiven context.

Remote sensing involves measurements on a targetwithout getting in contact with it and it comprisestechniques for collecting, storing, and processinggeoreferenced and geospatial data to extractvaluable information. In this context, data refer torepresentations stored in computer memory, which canbe manipulated using computers to derive meaningfulinsights. Remote sensing imaging systems primarilywork with georeferenced images, capturing Earth'ssurfaces, environment, atmosphere, etc. Theseimaging systems may be carried by satellites orairborne platforms like airplanes or drones. Forsatellite-based imaging, revolution of the satellitearound continuously rotating Earth allows periodiccapture of images over the same area. There are twomain types of imaging systems: passive and active.In passive systems, sensors detect reflected andemitted electromagnetic (EM) waves from Earth'ssurface, from mainly two types of energy sources,namely sunlight during the day and terrestrial heatat night. These sensors operate within specificspectral bands, converting energy into electricalsignals stored as two-dimension (2-D) images. Theprinciple is similar to optical cameras.Additionally, energy from Earth's thermal emissioncontributes to night time imaging, particularly inthe thermal infrared (IR) or far IR bands. Passiveremote sensing involves capturing images acrossvarious spectral bands, resulting in multispectraland hyperspectral images of specific regions onEarth.

In active imaging systems, microwave radar (RAdioDetection And Ranging) technology is utilized. Aradar transmitter emits a pulse of an EM wave with aspecific wavelength (in the microwave band). Whenthis pulse strikes a target, some of its energyreflects back to the radar antenna to which theradar receiver is connected. The receiver capturesinformation about the location and geometry of thetarget by recording the phase and amplitude of thereturned signal. By scanning the radar beam over anarea, an image of that region is formed. One of thelimitations of radar imaging is that, the size ofthe transmitting antenna is required to be large forobtaining images of high spatial resolution.

An artificial neural network (ANN) is a network ofneural nodes or perceptual nodes.1 In a feed forwardneural network, each node is fed with a weightedinput vector and the net sum of the weighted vectorsfrom several such nodes is passed through anonlinear function, whose response is the output ofthat node. The layers formed by the input nodes andthe output nodes are known as the input layer andthe output layer, respectively. The layers formed byother nodes are known as hidden layers. A network ofseveral such layers (along with input and outputlayers) forms an ANN, as shown in Fig. 9.1.2 Theconventional ANNs have very few hidden layers,usually not more than three. Using only one or twohidden layers is also common in many applications.In contrast, deep neuralarchitectures have relatively more numberof layers. Even more than 100 hidden layers is notuncommon. This is one of the distinguishedcharacteristics of deep neural networks (DNN) froman ordinary ANN.

The concepts used in deep neural computations aredecades older. In fact, they involve the same neuralnetwork computations as in a simple ANN model. Theseconcepts were introduced in 1980's and their basicprinciples still remain the same. However, a boom inusing deep architectures after almost three decadesof their proposition is mainly attributed to theadvancement in technology and science.

In images, some objects in the visual scene stand outfrom their neighboring or other regions grabbingimmediate attention of a human observer. Thoseobjects are called visuallysalient objects. The distinct perceptualquality of these objects compared to other objectsin the scene is called visualsaliency. This quality is attributed tothe behavior of an observer, and hence it issubjective. The mechanism by which the visuallysalient objects are selected is called visual attention. It is akey perceptual function in human visual system (HVS)for processing information from complex naturalscenes (Pinker, 1986). The visual attentionmechanism extracts essential features from redundantdata aiding the information processing in humanbrain. Our nervous system has a limited ability forsimultaneous processing of all the incoming sensoryinformation. The attention mechanism thusaccelerates this processing by selecting andmodulating the most relevant information. Thereexist multiple perceptive and cognitive operationsunder a hierarchical control process to establishglobal priorities to highlight some locations,objects or features in the visual field. A powerfulapproach to study visual saliency is to analyze eyegaze data of a viewer of a given scene, as thevisual saliency is associated with its correspondinggaze information of human beings. In this chapter wedevelop an understanding of visual attention,saliency and cognitive processing in the HVS.

5.1 Visual cognition

The way an individual acquires and processes the visualinformation is called visualcognition. It involves interpretation ofvisual sensation and identification of object, suchas, recognition of face, scene and object, visualattention and search, recognition of visual wordsand reading, control of eye movement and activevision, short-term and long-term visual memory,visual imagery, etc.

In the single view camera geometry it is not possibleto resolve the ambiguity of depth of the scene pointfrom its image point even with a calibrated camera,whose projection matrix is known. It is onlypossible to construct the ray in the 3-dimensional(3-D) world coordinate system passing through theimage point from the center of the camera. But in anoptical image a viewer can distinguish the objectpoint which forms the image. It is the opticalenergy received from that point in a surface of theobject and their distribution over its neighborhoodthat provide the key information to a viewer ininterpreting the object point and judging thedistances from the camera. In this chapter, wediscuss how the depth ambiguity in a single viewcamera geometry can be resolved with the help ofanother additional camera. The combined setup of twocameras is called stereo camera setup and thegeometry defined by them is referred to as stereo geometry.

12.1 | Epipolar geometry

A stereo setup consists of two cameras and a scenepoint, 𝑿, whose image is captured by the cameras,as illustrated in Fig. 12.1. The first (left) andthe second (right) cameras are specified by theircamera centers, 𝑪 and 𝑪′, respectively, whichcapture the images of the scene point on theirrespective image planes. In convention, the firstcamera or left camera is considered as the referencecamera of the stereo setup. By the rule ofprojection, let the image of the scene point, 𝑿, inthe first camera be 𝒙. Similarly, the image of 𝑿in the image plane of the second camera isrepresented by 𝒙′. The two images, 𝒙 and 𝒙′,correspond to the same scene point, 𝑿. Thus, forevery scene point, there will be two image points inthe image planes of the two cameras, if the scenepoint is visible from both the cameras.

Content based imageretrieval (CBIR) is a search techniquethat uses similarity of visual features to compareimages. It is also known as image based searchprocess, where, given a query image (with or withoutan accompanying text), the system provides a set ofimages that are similar to the query. This provisionis made available in most of the search engines,which enable us to search through the internet usinga query image and get several images that arerelevant to the query. The CBIR system has a lot ofapplications in various sectors, like education,research, tourism, health care, remote sensing, etc.In CBIR systems, while retrieving the resultsagainst a query, some domain specific information,like keywords, may also be provided to improve thequality of retrieval. The scope of such retrievalsystems could be extended to videos and multimediadocuments, which include text, audio, video,graphics, and images, as well.

Challenges and issues in building a CBIRsystem

Developing efficient CBIR systems is hurdled by severalchallenges. The image similarity computed forretrieving relevant images may not always satisfythe user's search intent. Often, objective criteriamay not fill the semantic gap in the representationof similar images. Some images, that are similar tohuman understanding, may be outright rejected asdissimilar images by a computational model. Manyobjective models are sensitive to noise and thepresence of a few outlier features may disturb thedecision by rejecting seemingly similar images oraccepting dissimilar images. In such cases ofsimilarity, thesystem does not explain why a pair of images aresimilar. Such sparingly occurring instances may beacceptable in some domains, but there are variousareas, like medicine and health care, where theverdict of a system is not acceptable without aproper explanation. Also, two images may be globallysimilar or they may have some local similaritybetween them. Capturing and localizing the localsimilarities for declaring a match between twoimages are also challenging. However, most of theCBIR systems work on global similarity.

Intensity images are limited in capturing surfacegeometry. Range imaging refers to an aggregation oftechniques to capture the surface topology as acollection of points that represent depths usingdifferent kinds of range sensors. Range images forma special class of digital images that requiredifferent processing techniques in their analysis.This chapter provides an overview of range imagingand processing.

13.1 | Range image

A range image is a 21/2-D or 3-D representation of the scene. It issometimes called 21/2 -D representation because itcaptures only the surface information of an objector scene as discretized points to represent it as animage. A range image, 𝑓(𝑖, 𝑗), records thedistance, d, to the corresponding scene point at(𝑥, 𝑦, 𝑧) for each image pixel, (𝑖, 𝑗), asshown in Fig. 13.1. In the array representation of arange image, the pixel values correspond to thedistances of the surface points, unlike conventionalRGB camera image pixels that represent intensities.The distribution of all the recorded values of dforms the functional distribution of the surfacepoints over the discretized space of the image,which is known as range data or depth data. Thisdepth information may also be represented as a setof 3-D scene points, also called a point cloud. Fora given functional value, 𝑓(𝑖, 𝑗), of an imagearray at an index position of (𝑖, 𝑗), thecorresponding pixel value of a scene point (𝑥, 𝑦,𝑧) in the discrete 3-D space is represented as (𝑖,𝑗, 𝑓(𝑖, 𝑗)). An example of a range image isshown in Fig. 13.2, where the intensity image iscaptured as an RGB color image and its correspondingrange image is captured using Microsoft™ Kinectsensor.