ALGORITHM FOR WIKIPEDIA SPIDER

Wikipedia lists can be used to identifyWikipedia articles about clerics. Inspired by Gong (n.d.), I implement an automated search through Wikipedia that simultaneously finds candidate entries and classifies whether each entry is the biography of a cleric. This methodology allows me to identify all of the Wikipedia articles about clerics. For this approach, I train a statistical text classification model to distinguish articles about clerics from other types of articles. I develop a program called a “spider” that moves from link to link within Wikipedia. The program starts by visiting a set of initial Wikipedia entries that I specify (I start with a list of approximately forty clerics from my data set). The program uses a statistical text classifier called a random forest (Breiman 2001) to classify each page it visits as “cleric” or “not.” For each of the Wikipedia entries classified as “cleric,” the program then follows every hyperlink from that entry, visits each of them, and repeats the same classification process on the resulting pages’ links. The program continues until all links in the resulting network have been visited.

The statistical text classifier is key to the success of this procedure so it is worth explaining a few of the details. The classifier is trained on a training corpus that I generated from the Wikipedia pages about clerics in my data set. I went to each of these pages, collected all of the links, and classified 727 outgoing links to other Wikipedia articles by hand as either pointing to an article about a cleric or not. I use the text of these hand-coded pages as the training set. This has some practical limitations – because the training set is derived from the entries of clerics, it is most accurate when the links it is classifying come from a cleric entry. However, if the classifier mistakenly classifies an entry as a cleric biography when in fact it is not, then the next set of articles that the classifier faces comes from a different distribution than the training set, making misclassification more likely.