Introduction

Students’ attendance and preparation for lectures often is voluntary, and assessment of learning outcomes occurs only at the end of the term. These structural challenges may induce students to focus both their attention and study efforts, largely on the final exam, instead of engaging continuously with the lecture’s content as the semester progresses. Especially in this situation, instructors are keen to foster learning, using innovative teaching tools and techniques, such as an audience response system (Gormley-Heenan and Mccartan Reference Gormley-Heenan and Mccartan2009; Mayhew Reference Mayhew2019) and flipped classrooms (Jenkins Reference Jenkins2015). However, the success of new teaching tools cannot be taken for granted, and research into the effect of educational technology has attracted ample scholarly attention (Bedenlier et al. Reference Bedenlier, Bond and Buntins2020; Henderson et al. Reference Henderson, Selwyn and Aston2017; Selwyn Reference Selwyn2016). This research has demonstrated how the informed use of new teaching methods can foster students’ engagement with courses and their material (Schindler et al. Reference Schindler, Burkholder and Morad2017).

Against this backdrop, we explore to what extent educational technology can counteract the structural challenges of large lecture courses. More specifically, can instructors, by providing students with regular unobtrusive encouragements to engage with the course and its literature improve learning outcomes? We address these questions using a pre-registered experiment, in which students receive weekly text messages, informing them about the upcoming lecture, and the assigned readings. Building on a literature researching the effect of recurring reminders to induce positive behavioural changes (e.g. Castleman and Page Reference Castleman and Page2015; Dale and Strauss Reference Dale and Strauss2009; Fjeldsoe et al. Reference Fjeldsoe, Marshall and Miller2009), we expect that students’ self-motivated learning and performance increases, when they receive personalised reminders.

This article has two goals. First, we advocate using experimental designs, for testing the effectiveness of new teaching techniques, and show how experimental designs can be implemented in a large lecture. To this end, we developed an experimental design that can serve as an easily implementable template to facilitate more experimental tests of learning tools in political science and beyond. Second, we test whether short information treatments delivered by text messages raise learning outcomes. In our experiment, we rely on voluntary subjects from a large introductory lecture course. We randomly sample students, who receive text messages, reminding them about the weekly reading load for the course. We test whether these reading reminders affect two outcomes, corresponding to formative and summative learning: quizzes at the beginning of lectures and grades in a final exam. In the experiment, we do not find differences between students, who received a reminder, and those, who did not for both learning outcomes. In the discussion, we explore several reasons for the null effect, including, attrition, statistical power, and the strength of information treatments. More important than these particular design features, these null findings stress the need to evaluate new teaching techniques rigorously, and when possible, experimentally.

Reminding students to learn

Students’ independent engagement with assigned coursework prior to classes is a cornerstone of their academic success (Barkley and Major Reference Barkley and Major2020; Hockings et al. Reference Hockings, Cooke and Yamashita2008), and is an absolute prerequisite for many contemporary forms of teaching, such as flipped classrooms (Jenkins Reference Jenkins2015; O’Flaherty and Phillips Reference O’Flaherty and Phillips2015). Hence, a crucial task that is particularly prevalent in large lectures is getting students to engage with the material independently. This independent motivation on behalf of the students, which refers to the idea that students are willing to invest their time and attention in a course, cannot be taken for granted (Ambrose Reference Ambrose2010: 68). An easily measurable aspect of independent student engagement is students’ completion of assigned readings. Researchers have been tracking the completion rate of course readings for decades and, while results vary, the overall conclusion is dire in the sense that most students fail to sufficiently engage with their reading assignments (Johnson 2019). The rate by which students completed their reading assignments, plummeted by almost 60 percentage points towards the end of the last millennium (Burchfield and Sappington Reference Burchfield and Sappington2000), and has ever since, remained relatively stable at about 25% (Baier et al. 2011; Clump et al. Reference Clump, Bauer and Bradley2004; Ribera and Wang 2015). This drop coincides with the expansion and wider societal reach of higher education, which diversified students’ reading skills, as well as their motivation to work towards a university degree (Baron and Mangen Reference Baron and Mangen2021).

Completing weekly coursework is, of course, not a goal in itself, but ultimately is meant to improve learning outcomes. Literature on learning typically distinguishes between two types of outcomes and assessment (Ambrose Reference Ambrose2010; Bloom et al. Reference Bloom, Hastings and Madaus1971). Formative learning refers to feedback, while students are still in the process of learning. It is typically focused on a particular task, and provides immediate evaluations for students and teachers. Summative assessments occur at the end of a course or program and “give a final judgement or evaluation of proficiency, such as grade and scores” (Ambrose Reference Ambrose2010: 193).

Bearing in mind low reading completion rates, and the importance of continuous engagement for learning outcomes, it becomes increasingly paramount for instructors to encourage student learning, and enhance their intrinsic motivation to engage continually with course materials. To do so, we combine weekly quizzes, administered using audience response systems (ARS), and a reoccurring encouragement by the instructor to read the required materials via personalised text messages. Particularly researched in the realm of public health, this sort of intervention has turned out to be effective in supporting smoking cessation (Free et al. Reference Free, Knight, Robertson, Whittaker, Edwards, Zhou, Rodgers, Cairns, Kenward and Roberts2011), weight loss (Patrick et al. Reference Patrick, Raab and Adams2009), and inducing more healthy life choices in general (Fjeldsoe et al. Reference Fjeldsoe, Marshall and Miller2009). Furthermore, its effectiveness has been tested in the social sciences, where it was shown to increase personal savings commitments (Karlan et al. Reference Karlan, McConnell and Mullainathan2016), vote turnout (Dale and Strauss Reference Dale and Strauss2009), and compliance with court fines (Haynes et al. Reference Haynes, Green and Gallagher2013). Within an educational context, Castleman and Page (Reference Castleman and Page2015) demonstrate how personalised text messages can help to induce students to sign up for university courses on time.

Although text messages may not be students’ “dominant daily mode of communication” (Lenhart 2012: 1) anymore, we believe that they are an appropriate channel of communication for this study’s target group. Importantly, mobile phones are generally not inundated by unwanted incoming SMS, rendering text messages an easily noticeable reminder. This contrasts with other communication channels like emails, where reminders could easily be mistaken for spam and hence, never rise to the recipient’s awareness. Moreover, text messages can be accessed at any time that is convenient for the recipient, which decreases the likelihood that the intervention is regarded as an obtrusive nuisance. More technically, text messages can easily be customised to directly address the recipient, and provide relevant information. In fact, studies that forgo the customisation of text messages tend to suffer from high rates of attrition (Fjeldsoe et al. Reference Fjeldsoe, Marshall and Miller2009).

We suggest that text messages can improve student motivation and learning via three distinct mechanisms. First, students usually take numerous courses throughout the semester, which are competing for time and attention. Hence, as the expected amount of coursework is often substantial, students may simply forget to devote sufficient time to a lecture during a week. Recurring reminders can help courses to receive the necessary weekly attention from students. Second, our reminder is designed to create a sense of urgency. Assuming that students show at least a decent a priori interest in the lecture’s topic, they might generally be inclined to commit to the assigned literature, yet decide to procrastinate. By reminding them that the next lecture is coming up soon, the reminder stresses that procrastination is not a viable option anymore. Lastly, if designed appropriately, a reminder can minimise costs on the side of the student. By including a link to the assigned literature in the weekly message, students can be directed to the online repository containing the readings. This facilitates access to the relevant material, and thereby decreases the necessary time investment to look up the literature individually. The used intervention is designed to appeal to all three suggested mechanisms. However, given that the ensuing analysis rests on only one treatment group, we are unable to isolate the potential individual effects.

A recent review on student assessment in higher education, and its relation to learning outcomes, offers two important considerations to this paper (Pereira et al. Reference Pereira, Flores and Niklasson2016). First, the effect of new teaching techniques and tools on both kinds of learning must be assessed, especially if used frequently. Second, most evaluations of teaching tools rely on interviews, questionnaires, and surveys, making it challenging to offer effective advice. Hence, we evaluate the effect of reminder text messages on learning outcomes, using the following experimental protocol, for having students engage with the readings prior to class, and providing targeted feedback during the lecture. While all students are asked to complete weekly readings before coming to lectures, only one half of them receive weekly text messages, reminding them about the coursework. At the beginning of each lecture, all students take a short multiple-choice quiz on an ARS (audience response system). The ARS provides students with instantaneous feedback about their knowledge at the beginning of the lecture. This targeted feedback on particular gaps gives students a formative assessment during learning, and affords students, the ability to improve within each lecture. Hence, the targeted feedback enriches the summative assessment at the end of the course, and enables us to continuously measure learning outcomes, as the semester progresses.

Research design

Our research design draws on a sample of BA students from a large introductory lecture course at the University of Konstanz in Germany, during the spring semester 2018. The course is a mandatory requirement for all political science students, but attendance to the weekly lecture is voluntary. Students are evaluated using a multiple-choice exam at the end of the term. Among the voluntary participants, students who were randomly assigned to the treatment group receive a weekly personalised text message, containing both a reminder about the next lecture’s topic, and a link to the assigned literature. We gauge the effect of these reminders through two measures of learning outcomes: students’ performance in weekly quizzes conducted at the start of the lecture, and the grade on the final exam.

The design is pre-registered at EGAP (Evidence in Governance and Politics).Footnote ¹ In this section, we lay out the experimental design. Besides our analysis of the pre-analysis plan, we also assess the message’s effect on weekly attendance in the lecture and participation in the final exam. These two tests are not pre-registered, but help us explore alternative outcomes in the discussion.

The recruitment of participants proceeded as follows: first, we communicate our research agenda with all BA-students enrolled in the course “Introduction to Comparative Politics”. We utilised the university’s student service platform (ILIAS), for those who were interested in our research for collecting all required information. A total of 145 students volunteered to participate. We documented their voluntary participation with a signed letter of consent that collected relevant information for carrying out the study. The study and the data collection were approved by the ethics board at University of Konstanz. Due to strict privacy requirements, we were unable to collect any additional information about students’ individual characteristics, such as age, socio-economic background or current GPA.

We use simple random assignment at the individual level, for placing students in a control and treatment group of comparable size (50:50). For balancing the distribution, we use the participant’s first name to determine their gender.Footnote ² We assigned 73 individuals to the treatment group, and 72 to the control group in R. Of those students, several failed to provide their letter of consent, resulting in 61 treatment units and 64 control units for the analysis. All data are anonymised.

Our dependent variable “individual learning outcome” is measured in two ways. First, we use an ARS to post short weekly quizzes about the lecture content to the participants at the beginning of each lecture session. Whereas most ARS report answers anonymously, the tool we employ (Socrative) enables us to assign responses to individual students, and count the number of correct answers. Thereby, we are able to measure the weekly learning outcome on an individual level. We use weekly and pooled responses for gauging the overall learning performance per student. Using ARS also allows the instructor to discuss answers to each quiz question during lectures and foster formative learning. Second, we rely on grades in the final exam at the end of the term as a measure of summative learning. The exam comprises of 45 multiple choice questions, which are graded on a 5-point scale, ranging from 1.0 (excellent) to 5.0 (fail).

While the lecture itself is voluntary, the final exam is mandatory for all students. Because students have to sign up for taking the final exam during the term, some students might drop out after attending several lectures. Instead of assuming that individuals in the treatment and the control group have the same turnout probability, we explicitly test this claim. We do not expect differences in participation between individuals receiving the reminder, and those who do not.

We were able to conduct eight weekly quizzes during the experiment period. In the first session of the class, we ran a test version of the quiz for the participants to understand the interaction with the Socrative platform. In the following weeks, students, regardless of their status in terms of survey participation or treatment, answered five multiple-choice questions at the beginning of each lecture. Students first entered their student ID, and then answered the questions. In the analysis, we matched the self-reported ID with the information in the participation list. Socrative also automatically reports the absolute and the relative count of right answers.

Across all waves and participants, we collected 319 quiz results and 125 exam grades. This amount was reached, because not all of the participants always attended weekly lectures (i.e., weekly compliance rates varied). Given that self-reported IDs might be inaccurate at times (e.g., transposed digits), we carefully checked for reporting errors in the weekly quizzes. Pooling the data of all waves together, we cumulatively collected 545 person-quiz observations, of which, 319 were reported by voluntary participants of the survey. For the final grades, we were able to match 125 individuals to the self-reported ones in the survey data.

Do regular reminders affect formative and summative learning?

Reminding students to do the required readings and to come prepared to class would be a simple and effective tool for formative learning. Our expectation was that, sending students an SMS by the instructor about the weekly reading, alters student behavior, and increases formative knowledge. We first tested this proposition on a weekly basis, and then across all weeks.

Table 2 and Fig. 2a depict the distribution of the proportion of the correct answers in each lecture. We apply the T-Test for two sample means first. For the first week, individuals receiving the reminder answer 0.33 more questions correctly than the controlled ones at the 95% significance level (p = 0.048). This result reaches a power of 0.78, given n = 90. In the sixth lecture, the positive treatment effect is 0.5 at a 90% significance level (p = 0.062), and reports the highest power among the lectures (0.84). However, for all other weeks, we do not find statistically significant differences between both groups. Since there are only eight waves, we are not able to draw conclusions, regarding whether the intervention effects wear out, is more periodical, or even reinforcing.

Table 2

Correct Answers in weekly quizzes

Student were asked five questions each week. Power analysis is executed with the empirical treatment size with one-sided test. Total number of observations is 319

Fig. 2

a Errorplots with standard deviation of formative and summative learning outcomes. Figure displays weekly counts of correct answers in weekly quizzes (higher score is better). b Errorplots with standard deviation of formative and summative learning outcomes. Figure exhibits differences in final grades (lower score is better)

We also detect no statistically significant differences, when comparing students’ quizzes across all lectures. If we count all correct answers over the term and compare both groups, we find that students in the treatment group on average answer 0.05 more questions correctly p = 0.28.Footnote ⁶ Again, a power analysis shows that reaching the estimated effect size of 0.06 in both tests, even though much lower than the conventional small size of 0.2, we would have needed about 6000 participants (given α = 0.05, β = 0.2). The null finding alludes that reminding students to read, does not lead to better performance in assessments of formative learning.

In a similar vein, we conducted ANCOVA to assess the treatment effect at the individual level. There barely exists any correlation or interdependence between the treatment status and the number of lectures as a continuous variable.Footnote ⁷ Here, we test whether the accuracy of quiz answers is affected by the timing of each lecture, presuming that the text reminder did not exist. The effect of the lecture does not confound the treatment effect. The F-statistics of the treatment status is reported at 0.40, with a significance level of 0.57. The power analysis suggests that given the significance level of 0.05 and other empirical settings, the power of the treatment estimate reaches 100%. This result suggests that the text reminder is likely to have no positive effect on the quiz performance, even as the total possible learning time increases.

Finally, we fail to measure a long-term retrieval effect of the reading reminder, when examining grades in the final. Figure 2b illustrates the distribution of the grades in both groups. The average performance of the treatment group is 0.14 lower than the control group, yet this difference is not statistically significant (p-value = 0.23).Footnote ⁸ The worse performance of the treatment group prohibits a power analysis of the intended one-sided test. We therefore focus on the two-sided hypothesis, regarding the difference in terms of the exam grades between the treated and the control groups. To reach the empirical Cohen’s d = 0.16, 1286 units are needed, given a 95% significance level and 80% power. Thus, our study is of low statistical power (0.10 given the treatment size). Overall, our analysis indicates that reminding students to read does not improve performance in a final exam.

Evidence of self-selection: participation in quizzes is positively associated with taking the final exam

As an ancillary investigation, which was not pre-registered, we assess whether regular attendance in the lecture and participation in weekly quizzes, results in a higher turnout for the final exam. We analyse treatment and control group separately. Tables 3 and 4 demonstrate the association between taking part in the exam, and in the quiz (measured as finishing at least for one quiz). In both the treatment and the control groups, these two patterns are far from independent of each other (p _treatment = 0.009, p _control = 0.000 in χ ²-test with degree of freedom = 1). Provided that participation in weekly quizzes has an impact on taking the final exam, the positive association in both treatment and control groups, is significant at 95%-level (p _treatment = 0.0003, p _control = 0.000, one-sided T-Test). This finding indicates that students, who regularly come to and participate in the lectures, are more likely to take the final exam. This relationship holds true in both groups, suggesting that this characteristic is balanced among the untreated and treated units.

Table 3

Joint distribution of participation forms of students in the treatment group

Table 4

Joint distribution of participation forms of students in the control group