A rational test taker must be unaffected by framing manipulations in exam instructions. The theoretical model proposed by Espinosa and Gardeazabal (Reference Espinosa and Gardeazabal2013) confirms this is the case by showing that two score equivalent rules must always result in the same response pattern. By contrast, we consider a model based on Prospect Theory where test takers’ reference points depend on the framing of the scoring rule. The framings proposed in our intervention are summarized in Table 1 (see the Experimental Design section for further details).

Table 1

Framings

	Initial points	Right	Omit	Wrong
Mixed	0	+ 1	0	$-\rho$
Loss	Maximum Score	0	− 1	$-(1+\rho )$

Let $U_i\left(x_j\right)$ denote the utility function of student i when receiving outcome $x_j$ from prospect j (item j). Without loss of generality, fix prospect j as the prospect being evaluated by the decision-maker. So, from now on, we refrain from using subscript j in the notation. Prospect Theory (Kahneman and Tversky Reference Kahneman and Tversky1979) assesses that decision-makers “perceive outcomes as gains and losses, rather than as final states” and “the location of the reference point, and the consequent coding of outcomes as gains or losses, can be affected by the formulation of the offered prospects” (Kahneman and Tversky Reference Kahneman and Tversky1979). According to these ideas, in our model students perceive each item as a potential gain or loss. In other words, their reference point depends on the assigned framing and corresponds to the expected score excluding the evaluated prospect.Footnote ⁵ According to this formulation, the argument of $U_i\left(x\right)$ under each framing corresponds to the values presented in Table 1. Similar models have been considered in a testing context by Budescu and Bo (Reference Budescu and Bo2015) and Karle et al. (Reference Karle, Engelmann and Peitz2019).

For $x\ge 0$ , we let $U_i\left(x\right)=u_i\left(x\right)$ where $u_i:R_{+}\to R$ is twice differentiable with $u_i\left(0\right)=0$ , $u_i^{\prime }\left(x\right)>0$ and $u_i^{\prime \prime }\left(x\right)\le 0$ . Following the widespread formulation by Kahneman and Tversky (Reference Kahneman and Tversky1979), for any $x<0$ let $U_i\left(x\right)=-{\lambda }_i{u}_i(-x)$ where ${\lambda }_i\ge 0$ is the loss-aversion parameter. A student is loss-averse if and only if ${\lambda }_i>1$ . This formulation implies that concavity in the gain domain becomes convexity in the loss domain (Kahneman and Tversky Reference Kahneman and Tversky1979, call this phenomenon the reflection effect). Throughout the paper, we measure concavity according to Arrow-Pratt measure $r_i\left(x\right)=-\frac{u_i^{\prime \prime }\left(x\right)}{u_i^{\prime }\left(x\right)}$ .

Let ${\tilde{p}}_i(k_i,z_i)$ be student i’s perceived probability of choosing the correct answer with $k_i$ denoting student i’s knowledge of the topic evaluated and $z_i$ accounting for other characteristics, such as self-confidence, that may influence student i’s perceived probability of answering correctly. We assume the perceived probability ${\tilde{p}}_i(k_i,z_i)$ to be independent of the particular scoring rule. To ease the exposition, we refrain from using the arguments determining the perceived probability and henceforth refer to it as ${\tilde{p}}_i$ .

In Prospect Theory probabilities are evaluated according to decision weights, which can differ from actual probabilities by overweighting small probabilities and underweighting moderate and large probabilities. Let ${\pi }_i^c\left({\tilde{p}}_i\right)$ and ${\pi }_i^w\left({\tilde{p}}_i\right)$ be the functions mapping student i’s perceived probability $\left({\tilde{p}}_i\right)$ into the decision weights of correct and incorrect answers, respectively (i.e., ${\pi }_i^x:{\tilde{p}}_i\in [0,1]\to [0,1]$ , $x\in \{c,w\}$ ). According to Prospect Theory (Kahneman and Tversky Reference Kahneman and Tversky1979), decisions weights are assumed to satisfy: i) ${\pi }_i^c\left(0\right)=0$ and ${\pi }_i^c\left(1\right)=1$ , ii) ${\pi }_i^w\left(0\right)=1$ and ${\pi }_i^w\left(1\right)=0$ , iii) ${\pi }_i^c\left({\tilde{p}}_i\right)$ is increasing in ${\tilde{p}}_i$ , iv) ${\pi }_i^w\left({\tilde{p}}_i\right)$ is decreasing in ${\tilde{p}}_i$ (i.e., increasing on $1-{\tilde{p}}_i$ ) and, v) ${\pi }_i^c\left({\tilde{p}}_i\right)+{\pi }_i^w\left({\tilde{p}}_i\right)\le 1$ . The latter assumption implies that the perceived probability of correct ( ${\tilde{p}}_i$ ) and incorrect ( $1-{\tilde{p}}_i$ ) answers can be simultaneously underweighted (i.e., ${\pi }_i^c\left({\tilde{p}}_i\right)<{\tilde{p}}_i$ and ${\pi }_i^w\left({\tilde{p}}_i\right)<1-{\tilde{p}}_i$ ) but only one of the two can be overweighted (i.e., either ${\pi }_i^c\left({\tilde{p}}_i\right)>{\tilde{p}}_i$ or ${\pi }_i^w\left({\tilde{p}}_i\right)>1-{\tilde{p}}_i$ ).

Under the Mixed-framing, correct answers will result in a gain of 1 (normalized) point, wrong answers in a loss of $\rho \in (0,1]$ points, and non-responses will receive zero points. So, a student is expected to provide an answer under the Mixed-framing if:

(1) $\begin{array}{c}{\pi }_i^c\left({\tilde{p}}_i\right)U_i\left(1\right)+{\pi }_i^w\left({\tilde{p}}_i\right)U_i(-\rho )\ge U_i\left(0\right)⟺{\pi }_i^c\left({\tilde{p}}_i\right)u_i\left(1\right)-{\pi }_i^w\left({\tilde{p}}_i\right){\lambda }_i{u}_i\left(\rho \right)\ge 0\end{array}$

Since ${\pi }_i^c\left({\tilde{p}}_i\right)$ and ${\pi }_i^w\left({\tilde{p}}_i\right)$ are increasing and decreasing in ${\tilde{p}}_i$ , respectively, the left hand side of the latter inequality is increasing in ${\tilde{p}}_i$ . Thus, we can define ${\overline{p}}_i^{\mathrm{Mix}}$ as the minimum value for which the above inequality holds (i.e., the unique value of ${\tilde{p}}_i\in [0,1]$ solving equation (1) with equality). Thus, ${\overline{p}}_i^{\mathrm{Mix}}$ represents the cut-off probability at which student i chooses to provide an answer under the Mixed-framing. Running comparative statics on ${\overline{p}}_i^{\mathrm{Mix}}$ we obtain the results in Lemma Footnote ¹.

Lemma 1

Let $U_i\left(x\right)=u_i\left(x\right)$ for $x\ge 0$ and $U_i\left(x\right)=-{\lambda }_i{u}_i(-x)$ for $x<0$ , where $u_i\left(0\right)=0$ , $u_i^{\prime }\left(x\right)\ge 0$ and ${\lambda }_i>0$ . Under the Mixed-framing, non-response is increasing in the loss attitude parameter ( ${\lambda }_i$ ) and in the concavity of $u_i(.)$ .

The proof of the lemma is in Appendix A. As ${\overline{p}}_i^{\mathrm{Mix}}$ is increasing in ${\lambda }_i$ and in the concavity of $u_i(.)$ , either loss-aversion and/or risk-aversion (in the positive domain) might be causing non-response in the Mixed-framing.

Under the Loss-framing, students are told they will start the exam with the maximum grade. Correct answers will result in no points loss, wrong answers in a loss of $1+\rho$ points and non-responses in a loss of 1 point. A student is expected to provide an answer if:

(2) $\begin{array}{c}{\pi }_i^c\left({\tilde{p}}_i\right)U_i\left(0\right)+{\pi }_i^w\left({\tilde{p}}_i\right)U_i(-1-\rho )\ge U_i(-1)⟺-{\pi }_i^w\left({\tilde{p}}_i\right){\lambda }_i{u}_i(1+\rho )\ge -{\lambda }_i{u}_i\left(1\right)\end{array}$

Similarly as before, we can define ${\overline{p}}_i^{\mathrm{Loss}}$ as the cut-off probability at which student i chooses to provide an answer under the Loss-framing.

Lemma 2

Let $U_i\left(x\right)=u_i\left(x\right)$ for $x\ge 0$ and $U_i\left(x\right)=-{\lambda }_i{u}_i(-x)$ for $x<0$ , where $u_i\left(0\right)=0$ , $u_i^{\prime }\left(x\right)\ge 0$ and ${\lambda }_i>0$ . Under the Loss-framing, non-response is independent from the loss attitude ( ${\lambda }_i$ ) and decreasing in the concavity of $u_i(.)$ .

The proof of the lemma is in Appendix A. In contrast to the Mixed-framing, under Loss-framing, non-response is unaffected by loss attitude ( ${\lambda }_i$ ). Loss-framing eliminates the asymmetry between gains and losses that exists under Mixed-framing. As a consequence, the loss attitude does not affect non-response under Loss-framing.

At first glance, the second part of Lemma Footnote ² might be surprising, as concavity is generally associated to a higher level of risk-aversion. However, according to Prospect Theory, this is only so in the gain domain. The reflection effect implies that “risk aversion in the positive domain is accompanied by risk seeking in the negative domain" (Kahneman and Tversky Reference Kahneman and Tversky1979). This implies that more risk-averse students in the gain domain, who are risk-seekers in the negative domain, should display lower levels of non-response under the Loss-framing.

Next, we compare the level of non-response under the two framings. As ${\overline{p}}_i^f$ represents the cut-off probability at which a student chooses to provide an answer under each framing $f\in \{Mix,Loss\}$ , a higher value indicates greater non-response, all else equal. By comparing the two cut-offs, we can obtain the following result.

Proposition 1

Let $U_i\left(x\right)=u_i\left(x\right)$ for $x\ge 0$ and $U_i\left(x\right)=-{\lambda }_i{u}_i(-x)$ for $x<0$ , where $u_i\left(0\right)=0$ , $u_i^{\prime }\left(x\right)\ge 0$ and ${\lambda }_i>0$ . The Loss-framing induces lower non-response if

$\begin{array}{c}{\lambda }_i>\frac{u_i(1+\rho )-u_i\left(1\right)}{u_i\left(\rho \right)}\end{array}$

The proof is in Appendix A. Proposition Footnote ¹ provides a sufficient condition for observing a reduction in non-response under the Loss-framing.

The left hand side of the expression in Proposition Footnote ¹ is increasing in loss-aversion, while the right hand side is decreasing in the concavity of $u_i(.)$ .Footnote ⁶ Thus, both loss-aversion and the concavity of $u_i(.)$ can contribute to observe less omitted questions under the Loss-framing. The first effect is a consequence of canceling-out the effect of loss-aversion under the Loss-framing documented in Lemma Footnote ². The second effect arises from the reflection effect, which makes individuals more willing to take risks when confronted with the Loss-framing. Moreover, for any degree of concavity of $u_i(.)$ , it is always possible to find a degree of loss-aversion that induces less omitted questions under the Loss-framing (see Figure 1 for a graphic illustration).

Proposition Footnote ¹ implies that mild conditions are sufficient for the Loss-framing to induce higher non-response than the Mixed-framing, as highlighted in the next corollary.

Fig. 1

Graphical illustration of Proposition Footnote ¹ according to an exponential utility function ( $u_i\left(x\right)=\frac{1-e^{-r·x}}{r}$ for $r\ne 0$ , $u_i\left(x\right)=x$ for $r=0$ ) with $\rho =0.25$ and ${\pi }_i^w\left({\tilde{p}}_i\right)=1-{\pi }_i^c\left({\tilde{p}}_i\right)$ . The X-axis represents the degree of (absolute) risk aversion r. The Y-axis represents the degree of loss-aversion $\lambda$ . The blue line shows the combinations of risk and loss attitudes for which ${\overline{p}}_i^{\mathrm{Mix}}={\overline{p}}_i^{\mathrm{Loss}}$ . The area shaded in light gray shows the combination of parameters making ${\overline{p}}_i^{\mathrm{Mix}}<{\overline{p}}_i^{\mathrm{Loss}}$ and the area shaded in dark grey the combinations making ${\overline{p}}_i^{\mathrm{Mix}}>{\overline{p}}_i^{\mathrm{Loss}}$

Corollary 1

Test-taker displaying simultaneously concavity of $u_i(.)$ and loss-aversion is sufficient to observe lower non-response under the Loss-framing than under the Mixed-framing.

The proof of the corollary is in the Appendix A. Corollary Footnote ¹ establishes a sufficient (but not necessary) condition for finding a positive treatment effect on response-rates. This sufficient condition is illustrated in Figure 1 where ${\overline{p}}_i^{\mathrm{Mix}}>{\overline{p}}_i^{\mathrm{Loss}}$ always holds for any combination ${\lambda }_i>1$ (loss-aversion) and $r_i>0$ (concavity of $u_i(.)$ ). Previous studies have shown that, although heterogeneous, the population displays both concavity in u(.) and loss-averse attitudes (Fishburn and Kochenberger Reference Fishburn and Kochenberger1979; Abdellaoui Reference Abdellaoui2000; Abdellaoui et al. Reference Abdellaoui, Bleichrodt and Paraschiv2007, Reference Abdellaoui, Bleichrodt and Haridon2008; Andersen et al. Reference Andersen, Harrison, Lau and Rutström2008; Booij and Van de Kuilen Reference Booij and Van de Kuilen2009; Harrison and Rutström Reference Harrison and Rutström2009; Gaechter et al. Reference Gaechter, Johnson and Herrmann2010; Von Gaudecker et al. Reference Von Gaudecker, Van Soest and Wengstrom2011), so we can expect the condition in Proposition Footnote ¹ to hold more frequently than the opposite. These observations provide the theoretical background for our main hypothesis.

Hypothesis 1

Average non-response will be lower under the Loss-framing than under the Mixed-framing.

It also follows from lemmas Footnote ¹ and Footnote ² that the reduction in non-response under the Loss-framing would be greater the more risk- and loss-averse the decision-maker is.Footnote ⁷ This implies that women who have been found to be more risk-averse (e.g., Eckel and Grossman Reference Eckel and Grossman2002; Fehr-Duda et al. Reference Fehr-Duda, De Gennaro and Schubert2006) and more loss-averse than men (e.g., Schmidt and Traub Reference Schmidt and Traub2002; Booij et al. Reference Booij, Van Praag and Van De Kuilen2010; Rau Reference Rau2014) might exhibit a higher decrease in terms of non-responses under the Loss framing.

Next, we address the consequences of Hypothesis 1 on test performance. Let $p_i\left(k_i\right)$ be student i’s actual probability of answering a specific item correctly.Footnote ⁸ If Hypothesis 1 is confirmed, the ratio of correct answers over the total number of questions must increase for any $p_i\left(k_i\right)>0$ .

The condition for observing an increase in test scores is more demanding due to the penalties for wrong answers $\rho$ . Additional answers increase the score if and only if $p_i\left(k_i\right)\ge \frac{\rho }{1+\rho }=\underline{p}$ . Let $A>1$ be the number of alternatives in a test item. For all the MCTs considered in our intervention $\rho \in \left(\frac{1}{A},,,\frac{1}{A-1}\right)$ . By replacing the values of $\rho$ by its highest value $\frac{1}{A-1}$ in the expression for $\underline{p}$ , we get that $\underline{p}=\frac{1}{A}$ . Note that $\frac{1}{A}$ is the probability of answering correctly by choosing a random alternative. Thus, if Hypothesis 1 holds, a sufficient condition for an increase in test scores under the Loss-framing is that the probability that the additional answers are correct is greater than if choosing randomly. If these conditions hold, Hypothesis 2 automatically follows:

Hypothesis 2

Average scores will be higher under the Loss-framing than under the Mixed-framing.

Finally, note that an increase in the proportion of correct answers is necessary but not sufficient for observing an increase in test scores.

4.1 Treatments

The experiment consisted of modifying the framing of the exam instructions according to the score equivalent rules in Table 1. The treatments only varied in the instructions, where two framings were used to describe the scoring rule:

• • Mixed-framing (control): Typical framing for a penalized MCT where each correct answer adds points to the score, omitted answers do not add or subtract points and wrong answers are penalized. Example:Footnote ⁹

  The exam is a multiple-choice test with 20 questions and 5 possible answers for each question. Only one of the 5 potential answers is correct. The maximum grade is 100 points. Correct answers give you 5 points. Each incorrect answer subtracts 1.25 points and finally each unanswered (omitted) question does not subtract or add points. For instance, a student who answered 16 questions correctly, left 3 unanswered questions and answered 1 question incorrectly, would have a final score of 78.75 over 100 (16*5- 3*0 - 1*1.25 = 78.75).

• • Loss-framing (treatment): We proposed a score equivalent manipulation of the Mixed-framing. Students were informed that they would start the test with the highest score. Correct answers would not add to or subtract anything from the initial score. Each wrong or omitted answer would decrease this initial maximum score by an amount equivalent to the one under the Mixed-framing. Example:

  The exam is a multiple-choice test with 20 questions and 5 possible answers for each question. Only one of the 5 potential answers is correct. The maximum grade is 100 points. You start the exam with a grade equal to this maximum score. The correct answers do not subtract anything. Each incorrect answer will subtract 6.25 points and finally, each unanswered (omitted) question will subtract 5 points. For instance, a student who answered 16 questions correctly, left 3 unanswered questions and answered 1 question incorrectly, would have a final score of 78.75 over 100 (100-16*0- 3*5 - 1*6.25 = 78.75).

4.2 Implementation details

We conducted the field experiment in 14 different sessions. Each session related to a different exam. Within each session, half of the students were randomly assigned to the Mixed-framing and the other half to the Loss-framing. All the exams took place during the 2018-2019 academic year.Footnote ¹⁰

Table 2 presents the main features for each of the sessions. All exams in our study were part of the official evaluation of three different courses (Introduction to Business, Human Resource Management, and Business) taught by eight different members of the Department of Business Economics.Footnote ¹¹ The exams lasted between 30 minutes and 1 hour. Stakes, penalty size, number of items and number of alternatives in each item varied slightly between exams and courses. Importantly, all were midterm exams accounting for between 20% and 33% of the final grade. None of these MCTs had a cut-off score or released material for the final exam. Thus, as in the model presented above, students should have been aiming to maximize their final scores.Footnote ¹²

All the students knew in advance that the exam was an MCT but they did not know the specific scoring rules. More importantly, students were not aware of the existence of different framings while doing the exam.Footnote ¹³ Each student participated in only one session and was only exposed to one of the two treatments.Footnote ¹⁴

Randomization was implemented in three different ways depending on organizational features of the exams. For computer-based exams, the on-line platform automatically and randomly assigned students to one of the framing conditions. In paper-based exams, hard copies of the grading instructions were delivered in such a way that immediate neighbors were assigned a different framing. This was done to ensure that the different framings were spread over the entire classroom to prevent the possibility that students’ seats were not random. Finally, in one of the courses, the treatments were assigned according to surnames in alphabetical order. Alphabetical order can be considered quasi-random. Since this course involved several sessions, to prevent surname effects, the mixed condition was implemented for the first half in alphabetical order in some sessions and for the second half in the remaining sessions.

For computer and surname-based randomization, whenever more than one classroom was available, students under the Mixed and Loss-framings took the exam in separate rooms. Students in these groups were assigned ex-ante (by the computer or their surname) to Mixed or Loss-framings and directed to take the exam in a particular room where all the other students were under the same treatment. Our aim was to avoid spillover effects. In case of taking the exam in a single classroom, an extra proctor was assigned to prevent spillovers between the different experimental conditions. Before starting the exam, students had 5 minutes to read the instructions (containing our treatments) and to privately ask any questions that they may have had regarding the evaluation method. After these 5 minutes, the exam started.

We also carried out a pilot study with 184 subjects from another course. In each exam, there were two shifts corresponding to different groups taking the course. The treatment was assigned at a group level. Despite the treatment being randomly assigned to each group, the group formation itself may not have been random. Therefore the observations from this pilot study are not included in our main results.Footnote ¹⁵

Finally, to gain better insights on the specific mechanisms driving the framing effect, we invited students to participate in an incentivized on-line survey. A total of 166 subjects who participated in the main study (30,9% of the total sample) filled in this survey. Participants were asked to complete 5 different incentivized tasks designed to measure their risk and loss preferences (see Appendix D for more information on the specific tasks). We present the survey and its results on Section 6.

Table 2

Description of the sessions/exams

Session	School	Year	Date	Num. of questions	Penalty size	Subjects	Stakes (%)	Instructor	Support	Randomization
0a	(Pilot)	2	Oct 2018	16	1/(A-1)	79	33	a	Comp.	Group
0b	(Pilot)	2	Oct 2018	16	1/(A-1)	105	33	a	Comp.	Group
1	Comp. Eng.+ Telecom. Eng.	1	Nov 2018	20	1/A	52	25	b	Paper	Seat
2	Electr. Eng.	1	Nov 2018	18	1/A	61	25	c	Comp.	Comp.
3	Comp. Eng. + Math & Telecom.	1	Nov 2018	20	1/A	41	25	d	Paper	Seat
4	Business	3	Apr 2019	25	1/(A-1)	39	20	e	Comp.	Comp.
5	Business	3	Apr 2019	25	1/(A-1)	43	20	e	Comp.	Comp.
6	Labor relationships	3	Apr 2019	25	1/(A-1)	34	25	f	Comp.	Comp.
7	Business	3	Apr 2019	25	1/(A-1)	56	20	f	Comp.	Comp.
8	Law & Business	3	Apr 2019	25	1/(A-1)	20	20	g	Comp.	Comp.
9	Tourism	1	May 2019	20	1/(A-1)	35	25	h	Paper	Alphabet
10	Tourism	1	May 2019	20	1/(A-1)	45	25	h	Paper.	Alphabet
11	Tourism	1	May 2019	20	1/(A-1)	42	25	i	Paper	Alphabet
12	Tourism	1	May 2019	20	1/(A-1)	27	25	i	Paper	Alphabet
13	Tourism& Business	1	May 2019	20	1/(A-1)	47	25	i	Paper	Alphabet
14	Tourism & Business	3	Apr 2019	25	1/(A-1)	12	20	g	Comp.	Comp.

Notes: Pilot was not within group randomized and is excluded from main estimations. Regarding randomization, alphabetical order is considered quasi-random but not-random. To prevent surname effects the treatment condition was implemented to first half of students in alphabetical order in sessions 10, 12 and to second half for sessions 9, 11, 13. In session 14 one student made a question referencing the grading system aloud compromising the validity of the data for that session

4.3 Data and descriptive statistics

Our main sample consisted of 537 students.Footnote ¹⁶ 266 students (49.53%) were assigned to the Mixed-framing and 271 (50.47%) to the Loss-framing. We observed their score in the test (Score), their total number of omitted questions (NR), the total number of correct answers (Correct), and the corresponding proportions (%NR and %Correct). We were also granted access to administrative data from the University of the Balearic Islands, including students’ academic record on a 0 to 10 scale (Acad. Rec.) and gender (Female). All data used in this study was conveniently anonymized by the IT services of the university.Footnote ¹⁷ To further check that the randomization worked correctly, we also retrieved information on test takers’ non-response from different computer-based MCTs other than the ones in the experiment (Non-Intervention %NR). These data were obtained from other exams performed during the 2018-2019 academic year and were available for 513 out of the 537 participating students.Footnote ¹⁸ We also constructed a pre-intervention non-response measure but in this case, we could only gather data for 427 students (80% of our sample).

Table 3 shows the overall average of our main variables (column 1) and the average for the Mixed-framing and Loss-framing (columns 2 and 3). It also shows the difference between treatments (column 4), standard errors (column 5), and the p-value for the two-sample t-test on means equality (column 6). Overall, Panel A in Table 3 shows no difference in gender composition or academic record between the students exposed to the Mixed and Loss-framings. More importantly, groups are also balanced in terms of non-response in tests outside the intervention, which can be considered a placebo test of our treatment (a proper placebo test is provided in Table B5 of Appendix B). Table B1 in Appendix B reports descriptive statistics by session. Though a few exceptions arise, treatment and control were balanced according to most of the observables at the session level. When presenting our results, we show they are robust when excluding sessions where any of the observables were not balanced between control and treatment. Taking all this together, we find support for our claim that randomization worked properly and that both groups are comparable ex-ante.

Table 3

Descriptive Statistics

	All	Mixed-framing	Loss-framing	Diff.	Std.error	p-value
Panel A: Control variables
Female	0.473	0.5	0.446	0.054	0.043	0.215
Acad. Rec.	5.78	5.793	5.768	0.026	0.120	0.831
Obs.	537	266	271
Non-Intervention %NR	0.127	0.127	0.127	0.001	0.013	0.950
Obs.	513	256	257
Pre-Intervention %NR	0.141	0.135	0.147	− 0.012	0.017	0.484
Obs.	427	214	213
Panel B: Outcome variables
NR	2.892	3.188	2.601	0.586	0.191	0.0023
Correct	12.739	12.857	12.624	0.234	0.329	0.4787
Score	5.089	5.178	5.001	0.177	0.158	0.2639
%NR	0.136	0.149	0.124	0.025	0.0089	0.0059
%Correct	0.590	0.594	0.585	0.01	0.0132	0.4736
Obs.	537	266	271

Female takes value 1 when the student is female and 0 otherwise. Acad. Rec. is the grade point average (GPA) of the student in the degree. Non-Intervention %NR is the average percentage of items omitted in other MCT different to the one of the intervention during the academic year 18-19. Non-Intervention %NR (Before the intervention) is the average percentage of items omitted in other MCT in the academic year 18-19 taking place before the intervention. NR (%NR) is the number (percentage) of omitted questions in the experiment. Correct (%Correct) is the number (percentage) of correct answers provided in the intervened exam. Score is the final grade in the intervened exam

Panel B in Table 3 presents the comparison between the Mixed and the Loss-framings for our main outcome variables. Raw averages show that non-response is significantly lower under the Loss than under the Mixed-framing both in total number (p-value=0.002) and as a percentage of the total number of questions in the exam (p-value=0.006). In other words, students under the Loss-framing answered more questions on average than students under the Mixed-framing. This finding is in line with our Hypothesis 1. By contrast, we find no evidence in favor of Hypothesis 2. When looking at the variable Score, we observe that the difference, although not significant, has the opposite sign to that predicted in Hypothesis 2. The same happens with the number and the proportion of correct answers.

In the next section, we present ordinary least squares (OLS) estimates of the treatment effect to provide a more accurate analysis by adding session-fixed effects and students’ controls. In what follows, results will be presented in terms of the non-response rate (% NR) but results are qualitatively the same by using the total number of omitted items.Footnote ¹⁹