Machine Learning with Python Principles and Practical Techniques

3.1 Need for Data Pre-processing

We live in an age where data is considered oil because we need data to train machine learning (ML) algorithms. The most important job for a data analyst is to collect, clean, and analyze the data and build ML models on the cleaned dataset. But often, the raw data that we obtain is noisy. It consists of many discrepancies, inconsistencies, and often missing values. To understand this situation, let us consider an example.

Suppose we have to predict the house price, and for this, we have collected data from a few previous transactions, as shown in Figure 3.1.

In a perfect situation, the captured data should be of this format, as shown in Figure 3.1. Here, we have the size of the house and the number of bedrooms as input features, while the price is the output attribute. We can predict the price of an unknown instance through regression.

But practically, in most situations, the captured data is not of good quality, and usually, we have a dataset, as shown in Figure 3.2.

You can see that this data is messy. There are a lot of unknown or missing values, and if we trained the model on this data, its prediction would be very poor. Also, you can identify the noise and incorrect labels like the second record price is incorrect and will result in poor model training.

We can also consider some more examples like if someone entered –1 in the “salary credited” column in the case of employee dataset. It does not make any sense and will be considered noise. Sometimes, we may have an unrealistic and impossible combination of data; for example, let us consider a record where we have Gender–Male and Pregnant–Yes.