Data cleaning is an iterative process. However, this chapter is more concerned with explaining the cleaning steps than the order in which you should do them. It reviews each data frame created in Chapter 4, performs a basic data exploration, and then cleans the “dirty” values, – those that have some anomalies caused during the process of importing the data into R or Python or because the original values were not ready to be imported into a different software. I focus on missing values, encoding, and, especially, symbols that were not imported correctly.

Dealing with Missing Values

Missing values are likely to be present in every data set. Data related to engineering activities can get lost, but, in general, a missing value indicates that a sensor did not record a particular data value. In social data, however, you will not only find an “NA” representing missing values but they may be coded with other values representing varieties of “missingness.” The coding of missing values is well covered by licensed software such as STATA or SPSS, but not explicitly covered in standard R or Python. As we learned in the last chapter, the memisc and foreign packages can import SPSS data very well into R (although there is less support from Python) and that Python and R can import without problems STATA files and the other formats we explored. Let's see how Python and R deal with missing data.

Identifying and Setting Missing Values: Codebooks

Finding a data set without missing values is rare. The problem is knowing whether the missing values are clearly identifiable or not. In general, if your data set comes with a codebook, then that means that you will know all the metadata needed. If you scrape the data, you may not know if you are missing values or not.

The SPSS data from ANES is a good example of the varieties of missing data in social science research. Table 5.1 from ANES time series for 2012 shows clearly which values should not enter into the calculations, but are nevertheless informative. That table was created using the codebook provided by ANES:

Table 5.1 has many values that should be considered missing values, and we have the codebook to identify them.