Skip to main content Accessibility help
×
Home
Hostname: page-component-6c8bd87754-xvt4m Total loading time: 0.329 Render date: 2022-01-17T20:52:12.194Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016)

Published online by Cambridge University Press:  31 December 2018

Simon Heuberger*
Affiliation:
Doctoral Candidate, Department of Government, American University, Washington, DC, 20016, USA. Email: sh6943a@american.edu
Rights & Permissions[Opens in a new window]

Abstract

Type
Letter
Copyright
Copyright © The Author(s) 2018. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

1 Introduction

This is the 2018 Political Analysis in-house replication of Muchlinski, Siroky, He, and Kocher (2016), henceforth MSHK. This work, “Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data,” was published in Political Analysis in Volume 24, Issue 1 in 2016.Footnote 1 It was accompanied by Dataverse replication material as required by the journal.Footnote 2 While this material was checked upon submission in 2015, recent replication efforts show that it does not support the claims made by MSHK.

Shown here specifically is that MSHK conducted in-sample predictions instead of out-sample predictions in their use of RandomForest as stated in the paper. RandomForest is a machine learning algorithm that constructs multiple decision trees to obtain more accurate predictions. The higher the number of trees in the forest, the higher the prediction accuracy. A RandomForest model needs to be fitted, or trained, on a data sample. This model can then be used to forecast, or predict, observations. If this prediction is made for an observation that is part of the training sample, it is an in-sample prediction. If this prediction is made for an observation that is external to the training sample, it is an out-sample prediction. By definition, predicting observations from the fitting sample based on a model derived from the same sample, i.e., in-sample predictions, will provide highly accurate results: We are predicting within the same sample that we trained on. To assess the viability of a RandomForest model, it is necessary to predict observations that were not used for the model fitting, i.e., to conduct out-sample predictions.

I am the current replicator for Political Analysis. I have been in this position since August 2017. I was not involved in the original assessment of MSHK’s submitted replication material. I walk through MSHK’s 2016 R code step by step. I start with the loaded source files, move on to model building, and finally address the out-sample analysis and insufficient output for Table 1. All R code, including comments and typos, is copied verbatim from material provided by MSHK. Some code lines in this replication analysis have been omitted for space reasons while others have been rearranged to fit page margins. These alignments do not affect the substantive content of the analysis.

2 Loaded Files

MSHK load three imputed .csv files: SambnisImp.csv, Amelia.Imp3.csv, and AfricaImp.csv. The first two are loaded as pre-imputed source files. The latter is imputed by MSHK in a separate R script. SambnisImp.csv is loaded into the R object data, which is further subset into data.full. Amelia.Imp3.csv is loaded into data2, myvars, and newdata. AfricaImp.csv is loaded into data3. These R objects are confusingly named, which makes the replication of the material more complex than it needs to be. While renaming them to separate them more clearly from each other would solve this, I have retained MSHK’s original names here in the interest of transparency. The R code to load the .csv files into the respective R objects is shown below.

3 Model Building

MSHK build four models. Three of these stem from previous studies: Fearon and Laitin (2003), Collier and Hoeffler (2004), and Hegre and Sambanis (2006). For each of these three studies’ models, they implement uncorrected and penalized logistic regression models. The fourth model is MSHK’s implementation of RandomForest. MSHK use these three external studies to showcase the superiority of RandomForest in predicting class-imbalanced civil war onset data. All models are trained on the R object data.full, which is a subset of SambnisImp.csv.

4 Out-Sample Analysis

After training the models, MSHK create three logit models for the external studies by Fearon and Laitin (2003), Collier and Hoeffler (2004), and Hegre and Sambanis (2006) as well as one model with RandomForest. All models load the R object data.full and are thus based on the imputed source file SambnisImp.csv.

Based on these models, MSHK make one prediction per model, turn the predictions into data frames, and subsequently set the seed to draw 737 random units from each predicted data frame. Each separate set of randomly drawn units is saved as a predictor object: predictors.rf for RandomForest, predictors.fl for Fearon and Laitin, predictors.ch for Collier and Hoeffler, and predictors.hs for Hegre and Sambanis.

MSHK then create confusion matrices with the predictor objects (based on the imputed source file SambnisImp.csv) and the variable warstds, which is a column from the data set data3, which in turn is based on the imputed source file AfricaImp.csv. Subsequently, MSHK load the R package ROCR. As per the ROCR package documentation, the function prediction() transforms the input data into a standardized format, while the function performance() calculates the area under the ROC curve if set to the parameter "auc", as MSHK do in the code shown below.

To sum up: For their out-sample analysis, MSHK create models based on SambnisImp.csv, make predictions based on SambnisImp.csv, draw random samples based on SambnisImp.csv, and calculate AUC scores based on SambnisImp.csv and one external variable based on AfricaImp.csv. In other words: MSHK conduct in-sample predictions, take random samples of these in-sample predicted probabilities, and compare those probabilities with true values from out-sample data. MSHK thus use the same sample to fit the model and conduct the predictions. This is not an out-sample prediction.

5 Output for Main Evidence

In their paper, MSHK provide Table 1 as the main evidence for their claim of the superiority of RandomForest. This table lists the predicted probabilities for civil war onset for 19 African countries and showcases the superiority of RandomForest over logit models in terms of prediction accuracy. MSHK provide CompareCW_dat.csv and identify it as the output that forms Table 1.

As the R code shows, CompareCW_dat.csv consists of the random predictor objects (predictors.rf, predictors.fl, predictors.ch, and predictors.hs) and the variable warstds. The predictor objects are based on the imputed source file SambnisImp.csv, while warstds is based on the imputed source file AfricaImp.csv. If we now juxtapose CompareCW_dat.csv and Table 1, we can see that it is not possible to match the information provided in the output .csv with the information listed in Table 1, as Figure 1 shows.

Figure 1. Actual Table 1 in Paper (left); View of Provided .csv in R (right).

CompareCW_dat.csv and Table 1 should show identical content. This is not the case. Table 1 consists of 19 rows, while CompareCW_dat.csv has 737. CompareCW_dat.csv does not have any identifiers that make the transition from this source file to the eventual Table 1 apparent and transparent. We do not know which predictor numbers correspond to which countries, as there is no information about the countries in the .csv file. Even if we assume that all instances where warstds == 1 sum up to the number of countries shown in Table 1, the numbers do not add up: There are 21 such instances in the .csv, but 19 in Table 1.

6 Conclusion

MSHK create several models (logit, RandomForest), make predictions based on these models, and draw random samples from these predictions. The data used for all of this comes from SambnisImp.csv. MSHK then create confusion matrices and calculate AUC scores based on data from SambnisImp.csv and one external variable from AfricaImp.csv. Rephrased in more generic terms, MSHK conduct in-sample predictions and take an in-sample sample, and then compare this in-sample sample with true values from out-sample data. Out-sample data only enters the equation in the final comparison, after the predictions have already been made with in-sample data.

In addition, the provided CompareCW_dat.csv cannot be compared to Table 1 because of its lack of identifiers. It is not possible to examine and verify the origin of the numbers in Table 1, which functions as the main piece of evidence in the paper.

Footnotes

Contributing Editor: Jeff Gill

References

1 Publicly available here: https://doi.org/10.1093/pan/mpv024.

2 Publicly available here: http://dx.doi.org/10.7910/DVN/KRKWK8.

Figure 0

Figure 1. Actual Table 1 in Paper (left); View of Provided .csv in R (right).

You have Access
2
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016)
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016)
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016)
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *