10.1 Support Vector Machines

Support vector machines (SVMs) are supervised machine learning (ML) models used to solve regression and classification problems. However, it is widely used for solving classification problems. The main goal of SVM is to segregate the n-dimensional space into labels or classes by defining a decision boundary or hyperplanes. In this chapter, we shall explore SVM for solving classification problems.

10.1.1 SVM Working Principle

SVM Working Principle | Parteek Bhatia, https://youtu.be/UhzBKrIKPyE

To understand the working principle of the SVM classifier, we will take a standard ML problem where we want a machine to distinguish between a peach and an apple based on their size and color.

Let us suppose the size of the fruit is represented on the X-axis and the color of the fruit is on the Y-axis. The distribution of the dataset of apple and peach is shown in Figure 10.1.

To classify it, we must provide the machine with some sample stock of fruits and label each of the fruits in the stock as an “apple” or “peach”. For example, we have a labeled dataset of some 100 fruits with corresponding labels, i.e., “apple” or “peach”. When this data is fed into a machine, it will analyze these fruits and train itself. Once the training is completed, if some new fruit comes into the stock, the machine will classify whether it is an “apple” or a “peach”.

Most of the traditional ML algorithms would learn by observing the perfect apples and perfect peaches in the stock, i.e., they will train themselves by observing the ideal apples of stock (apples which are very much like apples in terms of their size and color) and the perfect peaches of stock (peaches which are very much like peaches in terms of their size and color). These standard samples are likely to be found in the heart of stock. The heart of the stock is shown in Figure 10.2.

1.1 Introduction

In today’s data-driven world, information flows through the digital landscape like an untapped river of potential. Within this vast data stream lies the key to unlocking a new era of discovery and innovation. Machine learning (ML), a revolutionary field, acts as the gateway to this wealth of opportunities. With its ability to uncover patterns, make predictive insights, and adapt to evolving information, ML has transformed industries, redefined technology, and opened the door to limitless possibilities. This book is your gateway to the fascinating realm of ML—a journey that empowers you to harness the power of data, enabling you to build intelligent systems, make informed decisions, and explore the boundless possibilities of the digital age.

ML has emerged as the dominant approach for solving problems in the modern world, and its wide-ranging applications have made it an integral part of our lives. Right from search engines to social networking sites, everything is powered by ML algorithms. Your favorite search engine uses ML algorithms to get you the appropriate search results. Smart home assistants like Alexa and Siri use ML to serve us better. The influence of ML in our day-to-day activities is so much that we cannot even realize it. Online shopping sites like Amazon, Flipkart, and Myntra use ML to recommend products. Facebook is using ML to display our feed. Netflix and YouTube are using ML to recommend videos based on our interests.

Data is growing exponentially with the Internet and smartphones, and ML has just made this data more usable and meaningful. Social media, entertainment, travel, mining, medicine, bioinformatics, or any field you could name uses ML in some form.

To understand the role of ML in the modern world, let us first discuss the applications of ML.

16.1 Introduction to Artificial Neural Network

Neural networks and deep learning are the buzzwords in modern-day computer science. And, if you think that these are the latest entrants in this field, you probably have a misconception. Neural networks have been around for quite some time, and they have only started picking up now, putting up a huge positive impact on computer science.

Artificial neural network (ANN) was invented in the 1960s and 1970s. It became a part of common tech talks, and people started thinking that this machine learning (ML) technique would solve all the complex problems that were challenging the researchers during that time. But sooner, the hopes and expectations died off over the next decade.

The decline could not be attributed to some loopholes in neural networks, but the major reason for the decline was the “technology” itself. The technology back then was not up to the right standard to facilitate neural networks as they needed a lot of data for training and huge computation resources for building the model. During that time, both data and computing power were scarce. Hence, the resulting neural network remained only on paper rather than taking centerstage of the machine to solve some real-world problems.

Later on, at the beginning of the 21st century, we saw a lot of improvements in storage techniques resulting in reduced cost per gigabyte of storage. Humanity witnessed a huge rise in big data due to the Internet boom and smartphones.

In various applications of computer vision and imageprocessing, it is required to detect points in animage, which characterize the visual content of thescene in its neighborhood and are distinguishableeven in other imaging instances of the same scene.These points are called key points of an image andthey are characterized by the functionaldistributions, such as distribution of brightnessvalues or color values, around its neighborhood foran image. For example, in the monocular and stereocamera geometries, various analyses involvecomputations of transformation matrices such as,homography between two scenes, fundamental matrixbetween two images of the same scene in a stereoimaging setup, etc. These transformation matricesare computed using key points of the same scenepoint of a pair of images. The image points of thesame scene point in different images of the sceneare called points ofcorrespondence or corresponding points. Key points ofimages are good candidates to form such pairs ofcorresponding points between two images of the samescene. Hence detection and matching of key points ina pair of images are fundamental tasks for suchgeometric analysis.

Consider Fig. 4.1, where images of the same scene arecaptured from two different views. Though theregions of structures in the images visuallycorrespond to each other, it is difficult toprecisely define points of correspondences betweenthem. Even an image of a two-dimensional (2-D)scene, such as 2-D objects on a plane, may gothrough various kinds of transformations, likerotation, scale, shear, etc. It may be required tocompute this transformation among such a pair ofimages. This is also a common problem of imageregistration.

13.1 Implementation of k-means Clustering and Hierarchical Clustering

In the previous chapter, we discussed various clustering algorithms. We learned that clustering algorithms are broadly classified into partitioning methods, hierarchical methods, and density-based methods. The k-means clustering algorithm follows partitioning method; agglomerative and divisive algorithms follow the hierarchical method, while DBSCAN is based on density-based clustering methods.

In this chapter, we will implement each of these algorithms by considering various case studies by following a step-by-step approach. You are advised to perform all these steps on your own on the mentioned databases stated in this chapter.

The k-means algorithm is considered a partitioning method and an unsupervised machine learning (ML) algorithm used to identify clusters of data items in a dataset. It is one of the most prominent ML algorithms, and its implementation in Python is quite straightforward. This chapter will consider three case studies, i.e., customers shopping in the mall dataset, the U.S. arrests dataset, and a popular Iris dataset. We will understand the significance of k-means clustering techniques to implement it in Python through these case studies. Along with the clustering of data items, we will also discuss the ways to find out the optimal number of clusters. To compare the results of the k-means algorithm, we will also implement hierarchical clustering for these problems.

We will kick-start the implementation of the k-means algorithm in Spyder IDE using the following steps.

• Step 1: Importing the libraries and the dataset—The dataset for the respective case study would be downloaded, and then the required libraries would be imported.

• Step 2: Finding the optimal number of clusters—We will find the optimal number of clusters by the elbow method for the given dataset.

• Step 3: Fitting k-means to the dataset—A k-means model will be prepared by training the model over the acquired dataset.

• Step 4: Visualizing the clusters—The clusters formed by the k-means model would then be visualized in the form of scatter plots.

Color is a psycho-physiological property of humanvisual experiences when the eyes look at objects andlight. Color is not a physical property of thoseobjects or light, rather, it is the result of aninteraction between physical light in theenvironment and human visual system (Palmer, 1999).For processing color images, it is required todevelop an understanding on how colors arerepresented following human perception.

3.1 Light sources

A broad range of electromagnetic spectrum, shown inFig. 3.1, consists of electromagnetic waves rangingfrom very long wavelengths at radio waves to veryhigh frequency at gamma waves. A very narrowinterval in this spectrum, toward the higher end ofspectral frequencies, accounts for the visible raysand it is called the visiblespectrum. The light and colors that ahuman eye perceives relate to the frequencies ofwaves that fall under the visible spectrum. Apictorial representation of the correspondence ofwavelengths in the visible range of the spectrum todifferent perceived colors has been shown in Fig.3.1. There are seven distinguishable colors in thefigure, violet, indigo, blue, green, yellow, orange,and red, usually known in order of their increasingwavelengths by the acronym of VIBGYOR. The luminancesensitivity function that is shown as a curve inFig. 3.1 is a function of the wavelength. It isempirically observed that the sensitivity of thehuman visual system is maximum in the green zone ofthe visible spectrum. The luminance sensitivityfunction gradually decays toward violet (higherfrequencies) and red (lower frequencies) from thegreen zone, as shown in the figure by the whitecurve.

In Chapter 1 on introduction of image processing, imageformation in a camera has been briefly described.Consider the image in Fig. 10.1. As a basic rule ofprojection, for a given scene point, 𝑷 , a ray from𝑷 that passes through the center of projection, 𝑶,intersects the image plane at its image point, 𝒑.This is a mapping, 𝑷 → 𝒑, of a three-dimensional(3-D) scene point to its two-dimensional (2-D) imagepoint. This rule of perspective projection isapplied for getting the image point of any scenepoint, in general. This particular geometry is thebasis of projectivegeometry in our context.

10.0.1 | Real and projective spaces

Consider a 2-D space, where a point, 𝒑, is denoted bya pair of coordinates, (𝑥, 𝑦), as shown in Fig.10.2. Since it is a cartesian product in real axis,the 2-D space is also denoted as ℝ2, and the point𝒑 belongs to the 2-D coordinate space. Followingthe coordinate conventions, these coordinates aredefined corresponding to an origin, 𝑶, and twoperpendicular axes meeting at the origin, namely,𝑥-axis and 𝑦-axis. The considered projectivespace, although defined in a 2-D space, implicitlyincludes a 3-D space behind its definition. Forexample, though all the points in an image are in a2-D plane, they are related to 3-D points of a scenewhich are lying on the ray of projection. This isthe abstraction of a 2-D projective space. Considera 3-D space, as shown in Fig. 10.2. If a ray passesthrough the origin, 𝑶, and the considered point,𝒑, 𝒑 is said to be the representative of the ray.Every point in this projected plane represents aray. In this case, the set of projection points,each representing a ray or straight line passingthrough the origin, is known as a 2-D projectivespace, ℙ2.

A document is an object which is primarily meant forhuman reading. It is not limited to only text.Besides text, which is primarily for reading, it mayalso contain figures, diagrams, photographs, tables,charts, etc. Many of these auxiliary componentsfacilitate the reading experience. Document imageprocessing involves processing of images ofdocuments. Examples of documents comprise of scannedimages of printed or handwritten pages, photographsof documents, etc. In general, images that containdifferent kinds of reading materials are consideredas images of documents. In this context, the imageof a text displayed in the environment is also anexample of document, e.g., an image of a signboard.Such kinds of texts are referred to as scenetexts.

A few examples of images of different kinds ofdocuments are shown in Fig. 16.1. The document inFig. 16.1 (a) contains printed text and graphics.Fig. 16.1 (b) is an example of an official document,which is a typical format of an official purchaseorder. Every organization usually has some standardtemplate or format of each official form used inregular administrative routines, which belongs tothis particular category of documents.

A page of magazine is shown in Fig. 16.1 (c), where thegraphics and text are overlaid in a specific layout.In Fig. 16.1 (d), an example of scene text is shown,which is a photograph of a stone tablet describing ahistorical monument. The image shown in Fig. 16.1(e) is a scanned document of a handwritten page, anexample of writing of the famous Bengali poet andNobel laureate, Rabindranath Tagore (1861–1941).