1.1.1 The Pandas Package

While some other programming languages have tools for manipulating and analyzing data in tables, Python does not. Consequently, it is necessary to install additional software to work with dataframes effectively. In this Element, we will use a software package called Pandas for this purpose.

1.1.2 A Note on Formatting

Throughout this Element, the names of keywords, methods, functions, properties, and variables are presented in a monospace font. This is to distinguish elements of code from the language used to describe them. For example, in the sentence “import is used to import packages,” the first instance of “import” is presented in monospace to indicate that it is a Python keyword, while the second is a verb describing its use. Similarly, when a word like dataframe is presented as code (e.g., DataFrame), it refers to Pandas DataFrames, and when it is not, it refers to the concept of a dataframe more generally (i.e., a tabular data structure).

1.3.1 Fewer Things to Learn

Programming with dataframes involves using a small set of processes such as filtering and counting to perform complex manipulations of the underlying data. Tasks in CL can be accomplished by combining these processes.

1.3.2 Faster Development

With dataframes, many tasks can be accomplished in just a few lines of code. Writing less code means shorter development times, fewer opportunities to make mistakes, and less time spent debugging.

1.3.3 Chainable Output

Processes performed on dataframes often produce other dataframes. This allows functions and algorithms to be chained together so that one CL task (e.g., creating n-grams) can be chained directly into other tasks (e.g., counting values).

1.3.4 Memory Efficiency

Dataframes require less memory to hold large amounts of data than do other data structures. This allows analysts to work with larger corpora.

1.3.5 Execution Speed

Many of Pandas’ DataFrame tools are implemented in a way that makes them extremely fast in comparison to Python scripts that do not use dataframes.

1.3.6 Vector-Based Operations

Dataframes allow computation on entire vectors (i.e., a sequence or list) at one time – for example, multiplying each element in a sequence by a number without cycling through the sequence.

1.3.7 Transfer

A dataframe created in Python can be imported into other languages like R, or into spreadsheet software like Excel or Google Sheets. Similarly, the principles of dataframe programming transfer to other programming languages as well.

1.3.8 Interactivity

The combination of these characteristics makes dataframes an ideal datatype for working with Python’s interpreter interactively (through, e.g., iPython, Jupyter, or IDLE).

1.4.1 The Corpus of Online Registers of English (CORE)

The code in this Element makes use of the Corpus of Online Registers of English (CORE; Reference Biber and EgbertBiber & Egbert, 2018). The Element employs CORE for both technical and linguistic reasons. First, CORE has been used in dozens of peer-reviewed studies in register variation and natural language processing. It is an established corpus with an active user base. Using CORE in examples and making the corpus available for download as a dataframe ensure that readers will have a corpus they may use in their own research. Second, CORE includes texts from a range of online registers, and these registers can be treated as subcorpora. Consequently, procedures that typically require multiple corpora (e.g., keyword analysis) can be demonstrated with a single corpus. Additionally, this register diversification allows analysts to compare the use of linguistic features within or across situations of use. Basic descriptive statistics of the corpus are presented in Table 1.

The corpus has been converted to a Pandas DataFrame for the exercises in this Element. This dataframe contains seven columns described in Table 2.

The dataframe version of CORE can be downloaded at www.cambridge.org/Keller or https://sites.google.com/view/programming-for-cl/home.

1.4.2 A Note on Operating Systems

Python is platform independent; Python code can be executed on any computer with an operating system that has a Python interpreter. This means that while the code in this Element was written on a computer running Microsoft Windows, most of the code will work on MacOS as well. However, due to differences in the file structures of the two operating systems, some adjustment may be necessary.

2.14.1 Working with Lists

Consider the following code.

beginning = ‘It was a dark and stormy night‘
words = beginning.split()
words

[‘It‘, ‘was‘, ‘a‘, ‘dark‘, ‘and‘, ‘stormy‘, ‘night‘]

In this example, we first assign the value ‘It was a dark and stormy night.‘ to a variable we have named beginning. In the next line, we invoke the split() string method on beginning and assign the return value to a new variable, words. split() is a string method that takes the value of the string and splits it at every occurrence of one or more characters. By default, strings are split on whitespace characters (the characters created by pressing the space bar, return/enter, or tab). However, you may change this behavior by passing different characters into the method (by including them in the parentheses).

The return value of split() is a list containing all the substrings that resulted from the split. Since we had a space between each word, the resulting list contains all the words in beginning.

In Python, list variables are denoted using square brackets []. The elements in the list are separated from each other with commas. Further, in the preceding example, we can see that each element in the list is a str variable because they are enclosed in quotation marks. We can create a list literal using this same notation.

ending = [‘and‘, ‘they‘, ‘all‘, ‘lived‘, ‘happily‘,
‘ever‘, ‘after‘]
type(ending)

list

2.14.2 Joining Elements in Lists

Just as there is a str method that converts a str to a list, there is a complementary method that joins a list of str variables into a single string, join().

‘ ‘.join(ending)

‘and they all lived happily ever after‘Since this is a str method, we invoke it on the string. Whatever string we use here is interspersed among the elements of the list.

‘^_^ ‘.join(ending)

‘and ^_^ they ^_^ all ^_^ lived ^_^ happily ^_^ ever ^_^ after‘

2.14.3 Accessing Elements in Lists

We can access an element of a list with its index using square brackets.

words[4]

‘and‘

Note that and is the fifth word in beginning, not the fourth. In Python, lists are 0-based (or 0-indexed). This means Python starts counting the elements in the list at 0, rather than 1. The elements of words with their indices are shown in Table 3.

If you ask Python how many elements are in the list though, it will provide the expected answer.

len(words)

7

The len() function returns an integer for the number of elements in a list or dict, or the number of characters in a string.

2.14.4 List Slicing

You can access a sublist (or slice) of a list using square brackets with a : separating the index of the first element and the index of the last element in the desired sublist + 1. This is called list slicing. The return type is also a list.

words[2:4]

[‘a‘, ‘dark‘]

Note that the sublist contains only elements 2 and 3 of words, not 4. This is because Python is a 0-based language. Since we start counting at 0, we stop counting one number below the second index. Thus, the length of the returned list (2) is equal to the second index (4) minus the first (2). When the desired slice starts at the beginning of the list, the first element can be dropped.

words[:4]

[‘It‘, ‘was‘, ‘a‘, ‘dark‘]

When the desired slice ends at the end of the list, you can drop the second index.

words[2:]

[‘a‘, ‘dark‘, ‘and‘, ‘stormy‘, ‘night‘]

It is possible to access the last element in a list with -1. This is an effective way to get the last n elements in a list when you are not sure how long the list is.

words[-1]

‘night‘

words[-4:]

[‘dark‘, ‘and‘, ‘stormy‘, ‘night‘]

2.14.5 Appending to, Removing from, Copying, and Extending Lists

We can add elements to a list using the append() method.

words.append(‘.‘)
words

[‘It‘, ‘was‘, ‘a‘, ‘dark‘, ‘and‘, ‘stormy‘, ‘night‘, ‘.‘]

The append() method does not return a new list with the appended value. Rather, it modifies the original list and returns None.

new_list = words.append(‘.‘)
type(new_list)

NoneType

It is also worth noting that we have appended a period, a punctuation mark, and now the name of our list is a bit misleading as it contains both words and punctuation. It is important to keep in mind that the name of a list has no bearing on what it contains. Let us rename the list from words to tokens.

tokens = words.copy()
tokens

[‘It‘, ‘was‘, ‘a‘, ‘dark‘, ‘and‘, ‘stormy‘, ‘night‘, ‘.‘, ‘.‘]

Whoops! We have added two periods to the end of words (once with words.append(‘.‘) and once with new_list = words.append(‘.‘). We need to get rid of one of those periods. We will use the remove() built-in method of list variables. remove() looks for the first instance of an element in a list and removes it.

words.remove(‘.‘)
words

[‘It‘, ‘was‘, ‘a‘, ‘dark‘, ‘and‘, ‘stormy‘, ‘night‘, ‘.‘]

Now, we will copy words again and store the new copy in the variable tokens. While we are at it, we will use the extend function to add all the words from ending to the same list. The extend() function is a built-in method of list variables that takes a list as an argument and concatenates it to the end of the first list. All the values of the second list (the one passed into the method) are appended to the end of the list that extend() is invoked on.

tokens = words.copy()
tokens.extend(ending)
tokens

[‘It‘,
‘was‘,
‘a‘,
‘dark‘,
‘and‘,
‘stormy‘,
‘night‘,
‘.‘,
‘and‘,
‘they‘,
‘all‘,
‘lived‘,
‘happily‘,
‘ever‘,
‘after‘]

It is also possible to simply add the two lists together using the + operator.

tokens = words + ending
tokens

[‘It‘,
‘was‘,
‘a‘,
‘dark‘,
‘and‘,
‘stormy‘,
‘night‘,
‘.‘,
‘and‘,
‘they‘,
‘all‘,
‘lived‘,
‘happily‘,
‘ever‘,
‘after‘]

You may wonder whether it is possible to copy a list using the assignment operator =. While it is possible just to set tokens to equal words this way, doing so will link tokens to words so that any changes made to either will be reflected in the other. Essentially, we will have two names for the same data. If we want to copy a list as a new, separate entity, we must use the copy() list method invoked on the original list. Lists and dictionaries behave this way, but strings and integers do not. Again, there are good reasons for this related to memory management (described in the following subsection), but it can cause a lot of confusion. Variables that hold other variables often behave this way, while variables that hold only raw data often do not. The technical way to describe this behavior is to say that list and dict variables are mutable (their values can be manipulated in memory), while str and int variables are immutable (their values cannot be changed after they have been created).

2.14.6 Memory Management with Mutable and Immutable Types

Since str variables are immutable, you might wonder what is happening in a line of code such as this:

greeting = ‘hello‘ + ‘ ‘ + ‘world‘
greeting

‘hello world‘

Here, three strings are created in memory and then concatenated together. The concatenation is performed in a number of steps equal to the number of strings that are being concatenated minus 1 (2 in this case). First, ‘hello‘ and ‘ ‘ are concatenated. Since strings are immutable, Python cannot change the value of either of the original strings and instead creates a third – ‘hello‘. Then, a new string is created from the concatenation of ‘hello‘ and ‘world‘ and assigned to the variable greeting. This process ends up generating six strings, only one of which is retained (greeting). The others stick around in memory until Python disposes of them.

Doing the same thing with a list is more memory efficient.

greeting = ‘ ‘.join([‘hello‘, ‘world‘])
greeting

‘hello world‘

Here, three strings are created (‘hello‘, ‘world‘, and ‘ ‘), as well as the list to hold them. Then, they are joined and assigned to greeting; in total, five variables are created instead of six. With small examples like this, the distinction might not seem very important, but as the length of the greeting grows, the number of variables used in the list approach increases linearly while the number used in the string concatenation increases exponentially. With a four-word phrase, the number of variables generated by the list approach is six – one str for each word (4); one for the separator character ‘ ‘, one for the list; and one for greeting. The number generated by the str concatenation approach is 14 – one str for each word (4); one for each separating space (3); one for each step in the concatenation procedure (6); and one for greeting. Mutable types like lists and dicts thus provide a significant advantage over str variables for corpus linguists who work with multiple texts that are each many tokens long.

Finally, the mutability of list variables also explains why methods like append() return None instead of a new list. If append() returned a new list, all the values in the list would also have to be copied and assigned to the new list while the old one would just sit in memory until Python had a chance to dispose of it. This would entail significant memory overhead for appending just one value to the list.

2.14.7 Working with Dictionaries

As described in the preceding discussion, dictionaries are like lists in that they hold other variables but differ in that the order of items in a dict cannot be accessed through indexing or slicing. Rather, values (the dictionary equivalent of list elements) are accessed through their keys. Keys are typically strings or integers. There are no type restrictions on values.

Literals in a dict are created using curly braces, {}, and keys are separated from their values using colons :.

word_counts = {‘beginning‘: len(words)}
word_counts

{‘beginning‘: 8}

We can then access the value associated with a key using square brackets [].

word_counts[‘beginning‘]

8

We can also use a variable to access values as long as the variable is an immutable type.

text = ‘beginning‘
word_counts[text]

8

2.14.8 Adding Items to Dictionaries

There is no append() method for dictionaries. To add a new key/value pair, we simply access the dictionary with the new key in square brackets and assign the value to it with the assignment operator =.

word_counts[‘ending‘] = len(ending)
word_counts

{‘beginning‘: 8, ‘ending‘: 7}

We now have two items in our dictionary, separated by a comma. Each item is a key and value pair. Changing a value in a dictionary is accomplished in the same way as adding a new value.

word_counts[‘ending‘] = 9001
word_counts[‘ending‘]

9001

As you might guess, a dictionary’s keys must be unique. If we try to add a new value with an existing key, we will simply overwrite the original value for that key.

2.14.9 More on Accessing Items in Dictionaries

We can access a dictionary’s keys using the keys() method and its values using the values() method.

word_counts.values()

dict_values([8, 9001])

Note that the keys and values are returned as lists (they are in square brackets []). Like all lists, the order of their elements is fixed, and the ith key is the key for the ith value. This behavior is useful for converting dictionaries to lists. We can also access the items (key/value pairs) independently using the items() dictionary method.

word_counts.items()

dict_items([(‘beginning‘, 8), (‘ending‘, 9001)])

Here, the return type is a list of tuple variables (each item is enclosed in parentheses). Tuples are a datatype similar to lists, but immutable. We will not be using tuples much in this Element, since in most corpus linguistics tasks that would call for a list-like variable type, it is more efficient to use list, dict, or Series (a column in a dataframe).

2.14.10 Iterating over Lists and Dictionaries

Sometimes, it is desirable to go through every element of a list or dict one at a time. For example, suppose we wanted to test whether each element in a list of tokens is punctuation or a word. To do this, we can use a for loop and the built-in string method isalnum(), which tests whether each character in a string is a letter or number character. If all characters in the string are alphabetical or numeric, the method returns True, and if not, False.

“abcd1359”.isalnum()
# returns True because all characters in “abcd1359” are
# either alphabetical or alphanumeric

True

“?”.isalnum()
# returns False because “?” is neither alphabetical nor
# numeric

False

“abcd123?”.isalnum()
# returns False because at least one character in “abcd123?”
# is neither alphabetical nor numeric

False

If we want to apply this method to each element in tokens, we need to set up a loop to iterate through the elements in tokens and invoke the isalnum() function on each element.

for token in tokens:
        result = token.isalnum()
        print(result)

True
True
True
True
True
True
True
False
True
True
True
True
True
True
True

In the preceding code block, the loop is created using the syntax

for a in b:

where b is a list and a is used to refer to the elements in that list. The loop will execute once for each element in b, each time making a refer to the current element. The : indicates that we are beginning a nested block of code. Everything that comes after the : and is indented will be executed on each pass of the loop.

In the preceding case, the code inside the loop executes 15 times (once for each element in tokens). On the first pass through the loop, a refers to the first element of tokens (It). On the second pass, a refers to the second element (was), and so on. Each pass through the loop, the isalnum() method is called on token, and the return value is stored in the variable result. Next, result is printed to the console using the print() function. The upshot of all this is that we end up with a sequence of Trues and Falses printed to the console.

Let us modify the loop so that instead of just printing True or False, we store the result of isalnum() in a list for later use.

is_word = []
for token in tokens:
        result = token.isalnum()
        is_word.append(result)
is_word

[True,
True,
True,
True,
True,
True,
True,
False,
True,
True,
True,
True,
True,
True,
True]

Now, instead of printing to the console, we append each isalnum() result to a list which we create before the loop begins using

is_word = [].

We now have two lists with corresponding indices (tokens and is_word); since lists are ordered, the ith element in tokens is associated with the ith element in is_word. This will be useful later, but we can test that this is the case now. The seventh element in tokens should be the punctuation mark . and the corresponding element in is_word should be False.

tokens[7]

‘.‘

is_word[7]

False

3.6.1 Counting Values – Frequency and Normalized Frequency

Many common corpus linguistic tasks require counting the number of times a word occurs in a corpus. With DataFrame corpora, this may be done by invoking the value_counts() method on the column of the DataFrame we wish to count in.

counts = c.type.value_counts()
counts.head()

type

Name: count, dtype: int64

Here, the type column is accessed using c.type and then the values in it are counted using value_counts(). The value_counts() method returns a sorted Series, which we then assign to the variable counts. Note here that the index of this Series is not numerical (as the index of c is). Rather, the index contains all the categories from the type column. Another way to think of this is that the numbers produced by the value_counts() method are indexed by their category (in this case, the word type) rather than by a location in the corpus (which would not make sense). We can access the frequency for a specific word in the same way we would look up a value in a dictionary with a key.

counts[‘order‘]

19744

Here, we use square brackets to look up the frequency of order in counts and find that it occurs 19,744 times.

We may wish to normalize these counts to a standard number of words to facilitate cross-corpus comparisons. To do this, we pass into the value_counts() method the normalize=True argument and multiply by the integer we wish to normalize to.

normed_counts = c.type.value_counts(normalize=True) * 100
normed_counts.head()

type

Name: proportion, dtype: float64

Now we see that the occurs about 4.8 times per hundred words. The normalized frequency of any word type in the corpus can be found by looking up the word in the Series index.

normed_counts[‘order‘]

0.028232491997731507

Without realizing it, we have leveraged one of Pandas greatest advantages – the ability to do vector-based calculations. Here, we multiply a Series by a single value. Pandas automatically knows to multiply each element in the Series by 100 and return a Series of the same length as the original with the same index. Because Pandas’ vector-based operations are implemented with highly optimized code, they are extremely fast to execute. Vector-based multiplication with normed_counts takes less than a millisecond, while alternate approaches may take orders of magnitude longer.

Whenever speed is an issue, we should use vector-based calculations. Pandas can perform basic arithmetic operations (+, -, /, and *) as well as the modular division operator (%) and the equivalence test operator (==) this way. A wider range of mathematical operations can be applied to Series using functions in the NumPy package.

3.6.2 Exporting DataFrames and Series to Files or the Clipboard

normed_counts contains information about the frequency of word types in our corpus, but viewing the series on the console only displays its head and tail. The series is also lost when the script ends, or when the application we used to run the code closes. Accordingly, we will often wish to export DataFrames to files for later use. We can do this using the to_csv() method.

The csv (comma separated values) file format stores tabular data as plain text. When a table is stored as .csv, each column is delimited with a comma (or tab), and each row is delimited with a return character. All the data in the cells of a .csv file are stored as plain text.

The to_csv() method takes a string as its first argument – the path for the .csv file. In Windows, passing “data.csv” into to_csv() creates a file called data.csv in the current working directory (or overwrites it if the file already exists).

If you are exporting a DataFrame, you may add a header row with the names of columns to the .csv file by including the argument header=True after the string with the file path. If you are exporting a Series, however, you may want to pass header=False as the second argument.

Finally, since many tokens may be commas, we should use the tab whitespace character as the separator character. We can indicate this with the sep=‘\t‘ argument (note that \t refers to the white space indentation that is created when the tab key is pressed in a text editor or word processor).

To export our normed_counts Series to a file, we use

normed_counts.to_csv(path, header=False, sep=‘\t‘)

where path is a string with the location at which we wish to save the file.

Exporting a DataFrame to a file is not always necessary. At times, you may want to export your data to the system clipboard so it can be pasted into spreadsheet software, or imported to another programming language as a dataframe. To do this, use the to_clipboard() method. This method does not require a file path, but by default still formats the data on the clipboard as a .csv file. Therefore, the header=True or header=False and sep=‘\t‘ arguments are still necessary. Alternately, you may wish to export the file to the clipboard in Excel format. In this case, the argument excel=True should be included and the header and sep arguments omitted.

normed_counts.to_clipboard(header=False, sep=‘\t‘)

or

normed_counts.to_clipboard(excel=True)

3.6.3 Measuring Range

We may wish to find the number or proportion of texts that a word occurs in (its range), or we may wish to find the frequency of every word in every text in which it appears. Both can be accomplished with Pandas’ groupby() method. groupby() takes all the rows in a DataFrame and groups them together with other rows that share a value in one or more columns. We can then apply one of a limited range of methods to each group. These methods summarize relevant characteristics of the group. This is useful if we wish to treat word types or texts as units of analysis.

ranges = c.groupby(‘type‘).text.nunique()
ranges = ranges.sort_values(ascending=False)
ranges.head()

type

Name: text, dtype: int64

Here, we group the rows of the corpus by word type, access the text column of each group, and then count the number of unique values that occur in the text column for each group using nunique(). Since each group is a word type, the resulting Series lists the ranges for every type in the corpus. Then, we sort the Series using sort_values(). By default, this method returns a Series sorted in ascending order. However, we can tell Pandas to sort in descending order with the argument ascending=False. Finally, we assign this series to the variable ranges and display its head on the console.

The results indicate that the is the type with the greatest range (it occurs in 48,493 texts), while . occurs in 48,290. It may be more useful to know the proportion of texts that each word occurs in. We can learn this by dividing the range for each type by the total number of texts in the corpus – a number we can get by accessing the text column of the DataFrame and invoking the nunique() method.

n_texts = c.text.nunique()
ranges_prop = ranges / n_texts
ranges_prop.head()

type

Name: text, dtype: float64

3.6.4 Indexing and Filtering

So far, we have been operating on entire columns of a dataframe, but many CL tasks require looking only at specific words or words within a predefined set. With Pandas, there are a variety of ways to zero in on specific rows. The most straightforward of these is to index a DataFrame or Series using square brackets []. As with lists, the elements of a Series can be accessed by their indices as one might do with a dict. In what follows, we get the frequencies and ranges we have calculated thus far for the word type misinformation.

counts[‘misinformation‘]

262

normed_counts[‘misinformation‘]

0.0003746410506181956

192

This approach works because we have already created the Series with the appropriate counts. However, a more flexible way to learn about specific words is to use Pandas’ ability to select lines from a DataFrame that meet certain conditions and return only those lines as a new DataFrame. This is done using the loc property of DataFrames or Series with methods like eq() which returns True for every value in a Series that is equal to a value (eq() is identical to the equivalence test operator == used in the previous section). For example, we can find all rows in our corpus where the word type is misinformation using

rows = c.loc[c.type.eq(‘misinformation‘)]
rows.head()

loc is an indexer. It is not a method, but rather a property that works a lot like a Python dictionary. Just as we can look up a value in a dict by passing in a key, we can use loc to look up a row (the value) with the label of that row in the index (the key). The syntax for doing this is identical to the syntax for looking up a value in a dictionary.

c.loc[500]

will return the row (as a Series) with the label 500 in the index of the DataFrame.

c.loc[500]

Name: 500, dtype: object

Unlike dictionaries, however, the loc indexer is very flexible in the type of data that can be used to look up values. For one, we can look up a set of sequential rows in the DataFrame using the slice notation for lists – the lower bound, followed by a colon :, followed by the upper bound.

c.loc[500:505]

Note here that the behavior of loc with a slice differs slightly from the behavior of lists. The upper bound (505 in the preceding code sample) is included in the slice returned by loc, but would not be included in the same slice of a list. We can test this by taking a column of the DataFrame, converting it to a list, and slicing it.

c.type.tolist()[500:505]

[‘do‘, ‘not‘, ‘have‘, ‘to‘, ‘get‘]

The final row with the token drunk in the slice returned by loc is not included in the list slice.

In addition to single values and slices, it is also possible to give loc a list of row labels. The loc indexer will return a DataFrame with one row for each value in the set or list.

indices = [0, 5, 15, 20]
c.loc[indices]

Finally, we can give loc a list or Series of True / False values of the same length as the number of rows in the DataFrame and loc will return all the rows that correspond to a True value.

# create a mini-corpus with just the five rows of the corpus
# DataFrame
mini_corpus = c[:5]
# create a list with five True / False values.
indices = [False, True, False, False, True]
# loc the indices that correspond to True in the indices list
mini_corpus.loc[indices]

This last approach to using loc is the key to locating rows that meet certain conditions. Because we can very easily get Series of True / False values using the eq() method, we can quickly locate all rows that meet the condition. Now we can see why

c.loc[c.type.eq(‘misinformation‘)]

finds all the rows of the DataFrame where the value of the type column is misinformation:

c.type.eq(‘misinformation‘)

returns a Series of True or False values and c.loc[] returns all the rows that correspond to True in that Series.

It is possible to use this approach to apply multiple conditions. If we want to extract only those rows where the type column is equal to misinformation and the register column is equal to ne (news), we can do so by including multiple logical tests inside the square brackets [] after loc.

rows = c.loc[c.type.eq(‘misinformation‘) & c.register.eq
(‘ne‘)]
rows.head()

Each condition must be joined with & to indicate and, or | to indicate or. A ~ can be placed in front of a condition to mean not. The following code locates instances of misinformation from registers other than news.

rows = c.loc[c.type.eq(‘misinformation‘) &
~c.register.eq(‘ne‘)]
rows.head()

3.6.5 Selecting Rows before or after Rows that Meet a Condition and Removing Unused Categories

Often, we may search for a word not because we are interested in the word itself, but rather what comes before or after it. We can do this by rolling the dataframe forward or backward using

shift(n)

where n is an integer. shift(n) moves the content of the DataFrame forward n rows such that row i becomes row i + n. Consider the examples below.

c.head()

c.shift(1).head()

After shifting the dataframe forward 1 row, row 0 becomes row 1, 1 becomes 2, 2 becomes 3, and so on. Row 0 is populated with null values (NaN). DataFrames can also be shifted backward by passing a negative integer into shift(). This makes row 1 become 0, 2 become 1, 3 become 2, and so on.

c.shift(-1).head()

This allows us to locate rows that come after or before rows that meet certain conditions. If we are interested in finding the set of words that come immediately after personal pronouns, we need only shift the DataFrame forward one row and find all rows with a personal pronoun tag (PRP) in the tag column.

df = c.loc[c.shift(1).tag.eq(‘PRP‘)]
df.head()

We can then get frequency counts for these words using value_counts().

df.type.value_counts().head()

type

Name: count, dtype: int64

If you inspect the end of the Series using tail(), you may notice that many of the counts produced by value_counts() are 0.

df.type.value_counts().tail()

type

Name: count, dtype: int64

These word types never occur in rows immediately before rows with personal pronouns. They are included in the output because when value_counts() is invoked on a column of the category datatype, the method produces counts for all categories in the column even if the value is 0. To drop these excess categories, we can invoke the cat.remove_unused_categories() method of the type column before we invoke value_counts().

df2 = df.type.cat.remove_unused_categories()
df2.value_counts().tail()

type

Name: count, dtype: int64

In the preceding code we chain together several Pandas methods. First, we access the type column. Then we remove unused categories using cat.remove_unused_categories(). Then we count the remaining values using value_counts(). Finally, we display the tail of the dataframe using tail(). This is a natural way to manipulate Pandas DataFrames, but occasionally, the resulting code gets too long to be readable. In these cases, it is possible to break the command across several lines by enclosing it in parentheses () and indenting one level.

(
df.type
.cat.remove_unused_categories()
.value_counts().tail()
)

type

Name: count, dtype: int64

We may also use Python’s line continuation character \ between the chained methods to split a long sequence of methods across multiple lines.

df.type.cat.remove_unused_categories().\
value_counts().tail()

type

Name: count, dtype: int64

3.6.6 Finding Words Using Wildcards and Regular Expressions

You may wish to filter a DataFrame for tokens that start or end with a certain letter sequence or that contain a certain substring. Pandas provides support for these tasks through the str module, which can be invoked through the . operator after the name of a column. In particular, three methods are useful here: str.beginswith(), str.endswith(), and str.contains(). The following code, for example, selects only those rows with values in the type column that begin with “un”.

un = c.loc[c.type.str.startswith(‘un‘)]
un.head()

Similarly, we might select rows with values in the type column that end with “-ing” and with values in the tag column that are verb tags.

ing_verbs = c.loc[(c.type.str.endswith(‘ing‘)) &
 (c.tag.str.startswith(‘V‘))]
ing_verbs.head()

In these examples, we passed strings into Pandas str methods (str.startswith(), str.endswith()). It is also possible to search for values that match a regular expression pattern using str.contains(), as you can see in the next example.

c.loc[c.type.str.contains(‘^un.+ing$‘)]

[15076 rows x 7 columns]

The string

‘^un.+ing$‘

is a regular expression pattern indicating the sequence un at the start of a string (^un), followed by one or more occurrences of any character (.+), followed by the sequence ing at the end of the string (ing$). The results include the range of words that fit this pattern.

A full treatment of regular expressions is outside the scope of this Element, but it is worth noting here that the behavior of Pandas’ string methods may not match expectations if your search term includes characters that have special meaning in a regular expression. For example, the ^, ., +, and $ characters in the preceding string are recognized as special characters. Respectively, they match the beginning of a string, any character, any number of repetitions from one to infinity, and the end of a string. Searching for them may produce unexpected results. If we want to find any type that includes a “.”, for example, we might be surprised if we use

str.contains(‘.‘)

c.loc[c.type.str.contains(‘.‘)].head()

Because the . character is used by the regular expression engine to match any character, Pandas matches every row that contains any character (i.e., all of them). To search for just rows where the type column contains a string with a . in it, we need to tell Pandas to treat the string argument for str.contains() as a string literal, not a regular expression pattern. We do this by passing in another argument, regex=False.

c.loc[c.type.str.contains(‘.‘, regex=False)].head()

3.6.7 Finding Any Word in a List

Suppose you want to search for any one of a set of words. For example, we may be interested in contrasting verbs with similar meanings, but different argument structures such as fill, pour, and load. We can instruct Pandas to filter the DataFrame for these words using the isin() method with a list of values to search for as the argument.

lemmas = [‘fill‘, ‘pour‘, ‘load‘]
c.loc[(c.lemma.isin(lemmas)) &
(c.tag.str.startswith(‘V‘))].head()

3.6.8 Finding Multiword Strings

It is possible to search for sequences of values using shift(). We previously used shift() to find rows that came before or after rows that met certain conditions. We can also use shift() to apply conditions to multiple rows. In this way we can test whether the type in row n is the first word of a multiword sequence, the type in row n+1 is the second word in a multiword sequence, and so on. If, for example, we want to look for the string i used to, we can search using the following conditions:

c.type.eq(‘i‘)
c.shift(-1).type.eq(‘used‘)
c.shift(-2).type.eq(‘to‘)

This will identify rows where the type column equals i, the type column in the next row equals used, and the type column in the row after that equals to.

rows = c.loc[c.type.eq(‘i‘) &
  c.shift(-1).type.eq(‘used‘) &
  c.shift(-2).type.eq(‘to‘)]
rows.head()

Python finds several instances of the sequence i used to, but returns only the first row in each sequence. This is because the unshifted value in our search term is the first word in the sequence. We can get a sense of the wider linguistic context for these sequences by constructing a context window for each hit (context windows are discussed in the next section), but there are times when we are interested instead in the possible values of a variable slot in a multiword sequence. If we are interested in learning, for example, which words can fill the * position in the lexical bundle on the * hand, we can search for rows with on in the type column two rows before (using shift(2)), the one row before (using shift(1)), and hand one row after (using shift(-1)).

df = c.loc[c.shift(2).type.eq(‘on‘) &
                       c.shift(1).type.eq(‘the‘) &
                       c.shift(-1).type.eq(‘hand‘)]
(
  df.type.
   cat.remove_unused_categories().
   value_counts()
)

type

Name: count, dtype: int64

Here, we did not set a condition on the unshifted row, so it is not referred to in the square brackets after loc – we want to retrieve all rows that come after on the and before hand. Python returns the DataFrame with those rows and we store it in df. Then we access the type column, remove unused categories, and count the values that are left.

The results indicate that on the other hand is mostly fixed in this corpus. The overwhelming majority of instances of on the * hand occurs with either other or one in the * position.

3.7.1 Constructing N-grams

Linguists interested in finding sequences of tokens may use Pandas methods to convert a dataframe corpus to n-grams, n-length sequences of tokens. This may be done using the cat() method in Pandas’ Series.str module. When invoked on a column of a dataframe, str.cat() returns the values of the column concatenated into a single string. The others= argument, however, can be used to concatenate the values of two columns together where each value is the concatenation of the values in the same row of the two columns. The sep= argument tells Pandas which character to place between the concatenated values. For example, we can use str.cat() to concatenate every word type with the part of speech tag in the same row.

type_tag_pairs = c.type.str.cat(others=c.tag, sep=‘_‘)
type_tag_pairs.head()

Name: type, dtype: object

With shift(), however, str.cat() can be used to concatenate sequential values in the same column.

c.type.str.cat(others=c.type.shift(-1), sep=“ ”)

concatenates each value in the type column with each value in the same row when c.type is shifted one rank backward. In other words, it concatenates each value in the type column with the value in the subsequent row. The sep= “ ” argument tells Pandas to place a whitespace character between each of the values that are concatenated together. As there is no next row for the last row in the DataFrame, the value in that slot is null (NaN).

c.type.str.cat(others=c.type.shift(-1), sep=“ ”).head()

Name: type, dtype: object

The others= argument usually takes a single Series, but if we pass into the method a list of Series, it will concatenate values from each. For example,

c.token.str.cat(others=[c.type, c.lemma, c.tag], sep=“ ”)

concatenates the value in the token column of each row with the values in the type, lemma, and tag columns of the same row.

c.token.str.cat(others=[c.type, c.lemma, c.tag], sep=“_”).
head()

0              I_i_I_PRP
1      ‘m_am_be_VBP
2       22_22_22_CD
3   and_and_and_CC
4           I_i_I_PRP
Name: token, dtype: object

We can use this procedure to concatenate multiple sequential values by passing in a list with multiple shifts of the same column.

c.type.str.cat(others=[c.type.shift(-1),
                                                    c.type.shift(-2),
                                                    c.type.shift(-3)],
                                                    sep=“ ”).head()

0        i am 22 and
1      am 22 and i
2     22 and i have
3   and i have never
4   i have never had
Name: type, dtype: object

In the preceding example, we concatenate each value in the type column with the values in the type column one, two, and three rows after it. Thus, the list we passed with others= was

[c.type.shift(-1), c.type.shift(-2), c.type.shift(-3)]

Manually creating a list of values to concatenate is fine, but tedious for longer sequences. We can create a compact list of values to concatenate using a list comprehension of the form

[c.type.shift(-i) for i in range(1, n)]

where n is the number of words we wish to include in the n-gram. The following code creates a list of three-grams.

n=3
shifted_cols = [c.type.shift(-i) for i in range(1, n)]
ngrams = c.type.str.cat(others=shifted_cols, sep=“ ”)
ngrams.head()

0      i am 22
1    am 2
2 and2  22 and i
3 and i have
4 i have never
Name: type, dtype: object

To create a list of four-grams, we need only change the value of n from 3 to 4.

n=4
shifted_cols = [c.type.shift(-i) for i in range(1, n)]
ngrams = c.type.str.cat(others=shifted_cols, sep=“ ”)
ngrams.head()

0                                                i am 22 and
1                                                am 22 and i
2                                           22 and i have
3                                     and i have never
4                                     i have never had                                                
…
69933602             or webpage . hide
69933603             webpage . hide link
69933604                                               NaN
69933605                                               NaN
69933606                                               NaN
Name: type, Length: 69933607, dtype: object

The last three values in the Series are NaN because Pandas cannot concatenate three consecutive values to the types in these rows. NaN values can be replaced using the NaN replace (na_rep) argument. However, as NaNs interfere with counting, we will drop them using

ngrams = ngrams.dropna()

Next, to count all the four-gram types in the corpus, we will use value_counts().

ngram_counts = ngrams.value_counts()
ngram_counts.head()

type
_ _ _ _                     14026.
i do not         10683
? ? ? ?             9508
i do not know       5834
i do not think       5747
Name: count, dtype: int64

If we want to know the frequency of a specific n-gram, we can look it up in the ngram_counts Series using the dictionary lookup notation described earlier.

ngram_counts[‘on the other hand‘]

2724

Alternately, we can use the ngrams series and loc[] to locate n-grams that meet specific criteria. For example, one of the most frequent four-grams in this corpus is . i do not (beginning with a “.”). We may wonder what typically comes after this. To get a list of all four-grams that start with i do not, we use loc[] with str.startswith().

idonot = ngrams.loc[ngrams.str.startswith(“i do not”)]
idonot.value_counts().head()

type

Name: count, dtype: int64

Earlier, we used shift() to examine the potential variable slot lexical bundle on the * hand. We can use str.contains() and a regular expression for the same purpose.

ngrams.loc[ngrams.str.contains(‘^on the .+ hand$‘)].\
value_counts()

type

Name: count, dtype: int64

3.7.2 Context Windows

Many corpus-linguistic analyses require accounting for the immediate linguistic context around tokens of interest. However, up to this point, we have dealt only superficially with context (for example, by finding and counting the set of word types that occur immediately after personal pronouns). More realistic use cases for dataframe corpora include analyzing words in context or finding lists of collocates for one or more node words. In these cases (and others), it is necessary to adjust our unit of analysis from single word tokens to the linguistic environments around them – from tokens to context windows.

A context window is a small slice of a dataframe containing a predefined number of rows, usually n rows before and after a token of interest. There are several datatypes we might use to represent a context window in memory; each window could be a list or Series of row indices, or even DataFrames themselves. An excellent choice, however, is Python’s range.

In Python, a range is a datatype like a list in that it contains multiple values, but it is more constrained in that it is defined by two values – one representing the lower bound of a sequence of integers, and the other representing the upper bound. Ranges are a useful datatype for representing context windows because context windows are always contiguous sequences of rows. Therefore, it is only necessary to store the lower and upper bounds of the window. Keeping a list of all the intermediary indices simply wastes memory. Ranges, irrespective of the size of the sequence of integers they cover, always take 48 bytes, but they can be indexed and sliced as if they contained the full set of integers. Additionally, ranges can be passed into the loc[] indexer without having to convert them to another datatype first. Consequently, ranges provide a fast, memory-efficient, and programmatically straightforward way to call up a context window on a corpus.

Ranges can be constructed as literals using the range() function.

range(10,15)

creates a range with a lower bound of 10 and an upper bound of 15. As with lists, ranges are zero-indexed. This means that the range just created includes the integers 10, 11, 12, 13, and 14, but not 15.

Therefore, if we want to create a context window around the row labeled 1000 with five rows before 1000 and five rows after 1000, we use

range(1000–5, 1+1000+5)

The extra 1 before the second argument is to offset Python’s zero-based indexing. If we use a variable (i) instead of the number 1000, we use

range(i-5, 1+i+5)

Note that in cases where i-5 is less than 0 or where 1+i+5 is greater than the number of rows in the DataFrame, some of the values in the range will not correspond to rows in the corpus. More on this in what follows.

If we have a list of indices we want to iterate over and create a context window for each, we can use a list comprehension.

indices=[10, 20, 30]
cws = [range(i-5, 1+i+5) for i in indices]

This is precisely the approach taken in the following code sample: first, we use the loc[] indexer to find all rows where the value in the lemma column is equal to misinformation and take the index of the resulting DataFrame. Then we use a list comprehension to create a range for each item in the index.

indices = c.loc[c.lemma.eq(‘misinformation‘)].index
cws = [range(index-5, 1+index+5) for index in indices]

We can test this by taking the first context window in cws and giving it to the loc[] indexer of our DataFrame.

c.loc[cws[0]]

What we have produced is akin to a vertical Keyword in Context display for misinformation, but with additional information about each word.

Unfortunately, this approach still requires a bit of tuning. As noted earlier, this code will cause an error if index-5 is less than 0 or 1+index+5 is greater than the number of rows in the DataFrame. In these cases, Python looks for row labels that do not appear in the index and cannot find them (e.g., -1 or 100001 in a 100,000-token corpus). We can avoid this problem by slicing the DataFrame instead of passing the range directly into loc[].

context_window = cws[0]
c.loc[context_window[0]: context_window[-1]]

Here, context_window[0] is the lower bound in the range; context_window[-1] is the upper bound. Including the : between them tells Python to slice the DataFrame instead of locating rows by their labels. This effectively avoids the problem.

More complex concordances can be created with a little more coding. We will also use context windows to create collocates lists. These procedures are discussed in Section 4.

Specifying the Desired Output

Our first step in designing an effective algorithm is to specify the desired form of the output. Consider a collocates list. We would reasonably want to know the following information about each word that co-occurs with a node word:

1. 1. raw frequency in the corpus,

2. 2. frequency of co-occurrence with the node, and

3. 3. a measure of strength of association between the two (e.g., Mutual Information, Z-score, logDice).

The output of this algorithm, therefore, might reasonably be a DataFrame where every row is one collocate, and each column is one piece of information about that collocate.

Specifying the form of the output of an algorithm requires choosing a unit of analysis (what the rows of a DataFrame are meant to represent). In the preceding collocates list example, the unit of analysis is the collocate type. In contrast, the unit of analysis for an algorithm meant to find the frequencies of a word in every text in the corpus might take the text as its unit of analysis (each row would correspond to one text in the corpus and each column to one type). The output of an algorithm meant to generate a concordance, on the other hand, would take the word token as its unit of analysis (each row would correspond to one occurrence of the word).

Specifying the Form of the Input

Given data of a certain form, an algorithm always produces the same result. However, given data of a different form (even slightly), the algorithm may break the execution of the program, or succeed, but produce a result that is different from that which was expected. Accordingly, every algorithm includes a set of assumptions about the state of the data that goes into the first step. These assumptions are sometimes referred to as the algorithm’s input conditions. Assuming the input is always a dataframe with rows representing tokens, and columns for token, type, text, lemma, tag, pos, and register reduces the need to specify input conditions considerably, but there are still times when we will want to modify the dataframe in some way before the processing begins, for example, by removing function words before generating a collocates list, or adding a column for semantic tags prior to a keyword analysis.

Creating a Sequence of Steps to Process the Input and Produce the Output

The next step is to construct a sequence of operations that will complete the algorithm. As noted previously, there are often many ways to approach corpus-linguistic tasks, but it is generally a good idea to aim for clear, readable code where every step has a single, easily identifiable purpose. This not only helps with debugging, but also makes your code accessible to others who might need to check your work.

An Outline for the Rest of this Section

This section contains five additional sections – one each on algorithms for creating concordances, finding and analyzing lexical bundles, creating collocates lists, finding keywords, and performing key feature analysis. They were selected for inclusion here because they demonstrate important concepts in dataframe programming and, when examined in sequence, build on the previous algorithms. The linguistic and programming focuses for each algorithm appear in Table 4.

Input

1. 1. Import pandas.

2. 2. Load the corpus.

3. 3. Set search terms.

4. 4. Set left and right context window sizes.

Processing

1. 5. Filter the corpus.

2. 6. Extract the indices of remaining rows.

3. 7. Get context windows around each index.

4. 8. Create a new DataFrame using the context windows.

Output

1. 9. Rename columns to reflect distance from the key word.

2. 10. Insert a lemma column at the start of the DataFrame.

3. 11. Sort the DataFrame on the lemma column.

4. 12. Export the DataFrame to an Excel spreadsheet.

And now in Python. We will search for instances of the semantically related words fill, pour, and load.

# 1. Import pandas
import pandas as pd
# 2. Load the corpus
c = pd.read_pickle(‘CORE.pickle‘)
# 3. Create a list with our search terms
query = [‘fill‘, ‘pour‘, ‘load‘]
# 4. Create two integer variables to represent left and
# right context window sizes
L = 3
R = 3
# 6. Extract the indices of those rows
indices = c.loc[c.lemma.isin(query)].index
# 7. Get context windows around each index
context_windows = [range(ind-L, 1+ind+R) for ind in indices]
# 8. Create a new DataFrame using a list of slices of the token
# column of the corpus
slices = [c.loc[cw[0]:cw[-1]].token.values for cw in context_windows]
df = pd.DataFrame(slices, index=indices)
# 9. Rename the columns to reflect distance from the key word
df.columns = range(-L, 1+R)
# 10. Insert a column at the start of the DataFrame with the
# lemma of the key word
df.insert(0, ‘query‘, c.loc[indices].lemma.str.upper())# 11. sort the DataFrame on the lemma columndf = df.sort_values(by=‘query‘)
# 12. export the DataFrame to an Excel spreadsheet
df.to_csv(‘concordance.csv‘, sep=‘\t‘)

Most of the preceding code is covered in the previous section, though it may be helpful to zoom in on steps 6–9 to look at how df, the dataframe with the concordance, is created. In steps 6 and 7, a list of context windows for the terms in query is generated according to the procedure in Section 3. Then, in step 8, a list of slices is extracted from c.token corresponding to the context windows. This is done with a list comprehension, which iterates over every context window in context_windows and extracts the content of the token column in that window as a Series.

It is necessary to extract just the values to avoid errors caused by mismatched indices. The index of the returned Series is identical to the index of the section of the DataFrame the Series was extracted from. Therefore, each element in the list comprehension has a different index because each context window looks upon a different section of the corpus. Differences in the indexes would cause problems if we tried to put all the Series together into a single dataframe, as Pandas will try to align the columns based on their indexes instead of their distance from the query term. It cannot do this, however, as none of the indices overlap. Consequently, every word in every slice would get its own column and NaN values would be filled into all the empty cells. We avoid this problem by dropping the indexes from the slices using the values property. As the name suggests, values returns an array of values without any metadata – in this case, just the words in the token column without the index.

Also in step 8, slices is passed into pd.DataFrame() with the argument index=indices. This creates a new DataFrame where each column contains word tokens a certain distance from the query. The first column contains words that are L tokens to the left of the query. The second column contains items L-1 tokens to the left, and so on. To make this clear, we rename the columns in step 9.

In steps 10 and 11, we insert a column to record which lemma was in each context window and then sort the rows of the corpus based on that value. Finally in step 12, we write the concordance to a .csv file with tab characters delimiting columns.

To view the concordance, open the .csv file. If you would like to check that the algorithm produced the expected results, view the head of the output dataframe on the console.

df.head(3)

The algorithm in this section demonstrates how to create a concordance. One benefit of the output being a dataframe is that it can be filtered, sorted, and otherwise manipulated using the methods described in Section 3. However, examining a concordance is often a step that comes after previous analyses that indicate which linguistic features to examine in the first place. In the next sections, therefore, we will explore algorithms that identify prominent lexical bundles and collocates for a target word. Both algorithms identify linguistic features that might be examined in more detail through a concordance.

Input

1. 1. Import pandas.

2. 2. Load the corpus and create a subset of interviews.

3. 3. Set constants.

Processing

1. 4. Create an n-gram column and add it to the corpus.

2. 5. Create a DataFrame of four-grams with columns for

   1. 1. frequency, and

   2. 2. text range.

3. 6. Filter out n-grams that contain punctuation.

4. 7. Filter out n-grams that do not occur at least 40 times per million words.

5. 8. Filter out n-grams that do not occur in at least five texts.

Output

1. 9. Sort the DataFrame on frequency and range.

2. 10. Write to csv.

3. 11. Check output with head().

And now in Python. Note that the code comments are sparser than in the concordance algorithm.

# 1. import pandas
import pandas as pd
# 2. load corpus and extract interviews (‘it‘)
c = pd.read_pickle(‘CORE.pickle‘)
c = c.loc[c.register.eq(‘it‘)]
# 3. set constants
min_freq = 40
min_range = 5
punct = r‘[.,?!]‘
# 4. add four_gram column to corpus
shifted_cols = [c.type.shift(-i) for i in range(1, 4)]
c[‘four_gram'] = c.type.str.cat(others=shifted_cols, sep=‘ ‘)
# 5. create DataFrame with frequency and range information
df = pd.DataFrame({
‘freq‘: c.four_gram.value_counts() / (c.shape[0]-3),
‘range‘: c.groupby('four_gram').text.nunique()})
df.freq = df.freq * 1000000
# 6. filter ngrams that cross punc boundaries
df = df.loc[~(df.index.str.contains(punct))]
# 7. filter ngrams that occur fewer than 40 times per million
# words
df = df.loc[df.freq.ge(min_freq)]
# 8. filter ngrams that occur in fewer than 5 texts
df = df.loc[df.range.ge(min_range)]
# Output
# 9. sort df on freq, range
df = df.sort_values(by=[‘freq‘, ‘range‘], ascending=False)
# 10. write to .csv
df.to_csv(‘lexical_bundles.csv‘, sep=‘\t‘)
# 11. check output
df.head()

In the preceding algorithm, the four_gram column is created using str.cat() and shift(). Then a four-gram frequency dataframe is created in step 5. Frequency counts are normalized by dividing the raw frequencies of the four-grams by the number of four-grams in the corpus (equal to the number of tokens in the corpus – 3) and then multiplied by one million. The df is then filtered using the loc[] indexer, contains(), and ge(). The first two are described in Section 3, but ge() is new. The ge() function works exactly as eq(), but returns True for values that are greater than or equal to its argument.

The algorithm in this section demonstrates how to identify lexical bundles for a corpus, but the core of the procedure can be applied to a range of concepts where the unit of analysis is fundamentally a sequence of tokens. Indeed, n-grams provide a convenient starting point for phraseological analyses of multitoken units such as compounds and phrasal verbs, formulaic language, and verb-argument constructions.

Input

1. 1. Import Pandas and the log2 function from NumPy.

2. 2. Load the corpus and select only news texts.

3. 3. Create string to represent the node.

4. 4. Create integer variables to represent

   1. 1. left and right context window sizes, and

   2. 2. the base frequency cut-off and observed collocation frequency cutoff.

Processing

1. 5. Get frequency counts for all word types in the corpus.

2. 6. Get the indices of all instances of the node in the corpus using loc.

3. 7. Create context windows from the indices.

4. 8. Create a flat list of indices in all context windows (with duplicates).

5. 9. Create a Series with the frequencies of all word types that occur in the context windows.

6. 10. Calculate logDice.

Output

1. 11. Create a DataFrame with columns for base frequency, observed frequency of collocation, and logDice score.

2. 12. Filter out rows with frequency values below the cut-offs as well as the node (so it is not considered a collocate of itself).

3. 13. Sort the DataFrame by logDice scores in descending order.

4. 14. Export the DataFrame to a .csv file.

5. 15. Check the output with head().

And now the algorithm implemented in Python:

## INPUT
# 1. import pandas and the log2 function from numpy
import pandas as pd
from numpy import log2
# 2. load the corpus and select the tokens from texts of the
# news (ne) register
c = pd.read_pickle(‘CORE.pickle‘)
c = c.loc[c.register.eq(‘ne‘)]
# 3. create string to represent the node (node)
node = ‘misinformation‘
# 4. create integer variables to represent the context window
# and freq cut-offs
L = 5
R = 5
freq_cut = 5
coll_cut = 5
## PROCESSING
# Get Observed Frequencies
# 5. get frequencies of all word types
Y = c.type.value_counts()
# 6. get indices of node in the corpus
indices = c.loc[c.lemma.eq(node)].index
# 7. create context windows from the indices
cws = [range(ind-L, 1+ind+R) for ind in indices]
# 8. create a flat list of indices in all context windows (with
# duplicates)
cw_indices = [ind for cw in cws for ind in cw]
# 9. create a Series with the frequencies of all word types
# that occur in the context windows
O = c.loc[cw_indices].type.\
cat.remove_unused_categories().value_counts()
# 10. get log Dice
LD = 14 + log2((2*O) / (Y+Y.loc[node]))
## OUTPUT
# 11. create an Output DataFrame
df = pd.DataFrame({‘Y‘: Y,
                  ‘O‘: O,
                 
‘logDice‘: LD})
# 12. filter out rows: the node and words with frequency values
# below the cut-off
df = df.drop(node)
df = df.loc[df.Y.ge(freq_cut) & df.O.ge(coll_cut)]
# 13. sort the DataFrame by Log Dice score in descending order
df = df.sort_values(by=‘logDice‘, ascending=False)
# 14. export the DataFrame to an Excel file
df.to_csv(‘collocates of misinformation in news.csv‘, sep=‘\t‘)
# 15. check output
df.head(3)

The statement importing log2() from NumPy in step 1 begins with the from keyword. This allows us to import only a single function (or a small set of functions) instead of the entire library. The log2() function calculates base 2 logarithms and is used in step 10 to calculate logDice.

In step 5, a Series with frequency counts for every word type in the corpus is created. These are the Y values in the preceding formula. In steps 6–8, the indices of the node are extracted and used to generate context windows. A list comprehension is used in step 7 to flatten the list of indices in the context windows produced in step 6 – transforming it from a 2-dimensional list of lists of indices to a one-dimensional list of indices. The flattened list is then passed into loc[] in step 8, which returns a DataFrame with those rows that are in one or more context windows. Words that occur in multiple context windows are included multiple times. Then, value_counts() is used in step 9 to count all the values in the type column of the context windows around the node. This produces observed frequencies of co-occurrence for all types in all context windows and stores them in a Series called O. The cat.remove_unused_categories() in step 9 is necessary to remove word types from the index of the Series for those words that occur in the corpus, but not in a context window around an instance of the node. If we do not remove these, their observed frequencies of 0 will cause an error when Log Dice is calculated in step 10.

Log Dice is calculated using vector-based math operations according to the preceding formula. The entire Series O is multiplied by the scalar 2. The result is divided by the sum of the Series Y (corpus frequencies of all words) and the corpus frequency of the node (retrieved from the Series Y using Y.loc[node]). Finally, the result is logged using the log2() function imported in step 1. In step 10, 14 is added to the result of log2(). There is no need to add, multiply, divide, or log individual values. The operations +, *, \ and log2() all work on the entire Series at once.

Finally, in steps 11, 12, and 13, a DataFrame with frequencies, frequencies of co-occurrence, and logDice scores is created and filtered according to the frequency and co-occurrence cutoffs set in step 4.

The results indicate that the top collocate of misinformation in news is lies with a logDice score of 8.978. One might wonder if this is true in other registers. By changing ‘ne’ to ‘ob’ in step 2 (one might also want to change the name of the output file), we can find collocates of misinformation in opinion blogs. Results indicate that lies is also a collocate of misinformation in that register (logDice = 7.69), but the strength of association is not as great. The difference indicates that lies co-occurs with misinformation more than twice as often in news than in opinion blogs. On the other hand, the top collocate in opinion blogs is spread (logDice = 8.18), which has a logDice value of 7.2 in news. These differences may (or may not) point to differences in the discourse around misinformation in the two registers.

The algorithm demonstrated here is flexible enough to be extended to any corpus-linguistic analysis that relies on measuring association between linguistic items. Modifying one or more of steps 3 through 10 allows analysts to apply a different established measure of strength of association (e.g., Cohen’s d; z-scores), or test a new one. Alternately, strength of association can be calculated between different types of linguistic items (e.g., word type and part of speech tag; variable slot lexical bundle and slot fillers). Further, the output can be used to identify peculiar combinations of word types that may be investigated with close analysis of a concordance.