---
title: "Designing Teacher Performance Pay Programs: Experimental Evidence from Tanzania"
authors_and_venue: "with Isaac Mbiti and Youdi Schipper - Economic Journal 133(653), 2023"
venue: "Economic Journal"
year: 2023
doi: "10.1093/ej/uead010"
abstract: "We use a nationally representative field experiment in Tanzania to compare two teacher performance pay systems in public primary schools: a 'pay-for-percentile' system (a rank-order tournament) and a 'levels' system that features multiple proficiency thresholds. Pay for percentile can potentially induce socially optimal effort among teachers, while levels systems can encourage teachers to focus on students near passing thresholds. Despite the theoretical advantage of the tournament system, we find that both systems improved student test scores across the distribution of initial learning levels after two years. However, the levels system is easier to implement and is more cost effective."
pdf_url: "https://mauricio-romero.com/pdfs/papers/KF2.pdf"
canonical_url: "https://mauricio-romero.com/pdfs/papers/KF2.pdf.md"
source: research.qmd
note: >-
Machine-readable Markdown version of the paper, generated for LLMs.
---
**Designing Effective Teacher Performance Pay Programs:**
**Experimental Evidence from Tanzania**
Designing Effective Performance Pay
Isaac Mbiti1,2,5,6, Mauricio Romero3,2,∗ , and Youdi Schipper4
**Abstract:**
We use a nationally representative field experiment in Tanzania to compare two teacher performance pay
systems in public primary schools: a Pay for Percentile system (a rank-order tournament) and a "Levels"
system that features multiple proficiency thresholds. Pay for Percentile can (under certain conditions)
induce socially optimal effort among teachers, while Levels systems can encourage teachers to focus on
certain students. Despite the theoretical advantage of the tournament system, we find that both systems
improved student test scores across the distribution of initial learning levels after two years. However, the
Levels system is easier to implement and is more cost-effective.
**Keywords:** teacher performance pay, pay for percentile, incentive design, Tanzania
**Classification:** C93, H52, I21, M52, O15
*We dedicate this paper to the memory of our colleague Joseph Mmbando.*
∗**Correspondence address:** Av. Camino a Santa Teresa 930, CDMX, Mexico; Correspondence e-mail: mtromero@itam.mx
We are especially grateful to Karthik Muralidharan for his collaboration in the early stages of this project and subsequent discussions. We thank the leadership and staff at Twaweza for their collaboration and support on this project. We would also like to thank the editor (Sule Alan), three anonymous referees, Allison Bigelow, Austin Dempewolff, Mitch Downey, David Evans, David Figlio, John Friedman, Delia Furtado, Guthrie Gray-Lobe, Scott Imberman, Ronak Jain, Joseph Mmbando, Molly Lipscomb Johnson, Terence Johnson, Michael Kremer, Derek Neal, Omer ¨ Ozak, Bobby Pakzad-Hurson, Wayne Aaron Sandholtz, Enrique ¨ Seira, Daniela Scur, Jay Shimshack, Bryce Millet Steinberg, Tavneet Suri, Rachel Walet, and seminar/conference participants at
UC San Diego, Universidad del Rosario, NEUDC, PacDev, RISE, SOLE, SREE, and E-con of Education for their comments. Erin Litzow and Jessica Mahoney provided excellent research assistance through Innovations for Poverty Action. We are also grateful to EDI Tanzania for their thorough data collection and implementation efforts. The EDI team included Respichius Mitti, Phil Itanisia, Timo Kyessey, Julius Josephat, Nate Sivewright, and Celine Guimas. We received IRB approval from Innovations for Poverty Action, UC San Diego, and University of Virginia. The Tanzania Commission for Science and Technology (COSTECH)
also reviewed and approved the protocol. A randomised controlled trials registry entry and the pre-analysis plan are available at: [https://www.socialscienceregistry.org/trials/1009.](https://www.socialscienceregistry.org/trials/1009) Replication data is available at [Mbiti](#page-52-0) *et al.* [\(2022\)](#page-52-0).
1
# **1 Introduction**
Education systems in developing countries are typically characterised by weak accountability structures, which are often associated with low learning levels among students, limited observed levels of teacher effort, and inadequate institutional oversight and support for teachers [\(Chaudhury](#page-48-0) *et al.*, [2006;](#page-48-0) [Mbiti,](#page-51-0) [2016;](#page-51-0) [Bold](#page-47-0) *et al.*, [2017;](#page-47-0) [World Bank,](#page-54-0) [2018\)](#page-54-0). Because teachers play a central role in the education production function [\(Hanushek and Rivkin,](#page-50-0) [2012;](#page-50-0) [Chetty](#page-48-1) *et al.*, [2014b](#page-48-1)[,a\)](#page-48-2), and a large share of national education budgets are devoted to their compensation, policymakers and researchers are increasingly interested in interventions that increase teacher effectiveness through more robust accountability measures. Teacher performance pay programs are seen as a potential policy response to improve accountability because they strengthen the links between teacher remuneration and student learning outcomes [\(World Bank,](#page-54-0) [2018;](#page-54-0) [Bruns](#page-48-3) *et al.*, [2011\)](#page-48-3).[1](#page-1-0) Yet, the specific manner in which these programs link student performance to teacher pay varies greatly, ranging from simple proficiency threshold (or "bright line") designs to more complex value-added designs and rank-order tournaments [\(Imberman,](#page-50-1) [2015;](#page-50-1) [Breeding](#page-48-4) *et al.*, [2021\)](#page-48-4). However, there is limited evidence on how to best structure teacher incentives, especially in developing country contexts, which are less likely to have the requisite data management capacity to implement these schemes at scale [\(Breeding](#page-48-4) *et al.*, [2021\)](#page-48-4).
Incentive schemes based on proficiency thresholds are commonly used in education systems and have well-known weaknesses and strengths. Research from the teacher incentive literature and the broader school accountability literature suggests they tend to favour teachers who serve students from wealthier backgrounds, encourage teachers to focus on marginal students — which could exacerbate inequality in learning outcomes — and present challenges for policymakers who are tasked with selecting the appropriate (learning) thresholds that trigger rewards (or punishments) [\(Figlio and Loeb,](#page-49-0) [2011;](#page-49-0) [Neal,](#page-52-1) [2011;](#page-52-1) [Macartney](#page-51-1) *et al.*, [2021\)](#page-51-1). Despite these potential drawbacks, the widespread adoption of these systems suggests they may have some advantages compared to other incentive designs. In particular, threshold
1Teacher performance pay programs have been implemented in both developed and developing contexts. For instance, the share of US school districts with teacher performance pay programs increased from 7.9% in 2004 to 11.3% in 2012 (an increase of over 43%) [\(Imberman,](#page-50-1) [2015\)](#page-50-1). Less developed countries such as Brazil, Chile, and Pakistan have also implemented performance pay programs, often as large pilots [\(Alger,](#page-47-1) [2014;](#page-47-1) [Ferraz and Bruns,](#page-49-1) [2012;](#page-49-1) [Barrera-Osorio and Raju,](#page-47-2) [2017;](#page-47-2) [Contreras and Rau,](#page-48-5) [2012\)](#page-48-5).
(or "bright-line") designs may be well-suited for situations where the thresholds correspond to important objectives, such as curriculum goals. They also provide teachers with clear and salient targets. This can enable teachers to better react to the incentives and provides them with a clearer understanding of their performance [\(Brehm](#page-48-6) *et al.*, [2017\)](#page-48-6) — this feedback loop may be critical in contexts where teacher capacity is relatively limited (see [Bold](#page-47-0) *et al.* [\(2017\)](#page-47-0) for a review of teacher capacity in sub-Saharan Africa).[2](#page-2-0) Further, their transparency (and, thus, perceived fairness) could foster acceptance of the system among teachers [\(Fehr](#page-49-2) *et al.*, [2007\)](#page-49-2). A practical (and cost-saving) advantage is that these schemes require students to be tested only once and do not require panel data management to link beginning and end of year student performance. This arguably makes them better suited to developing country contexts where there is an urgent need to strengthen the public sector's data management capacity and promote a culture of using data in decision-making and implementation [\(World Bank,](#page-54-1) [2017a\)](#page-54-1).
Despite the popularity of incentive systems based on proficiency thresholds, insights from economic theory suggest that sophisticated teacher incentive designs, such as those based on rank-order tournaments, may induce greater — and potentially socially optimal — levels of effort among teachers than those based on proficiency thresholds [\(Lavy,](#page-50-2) [2009;](#page-50-2) [Neal,](#page-52-1) [2011;](#page-52-1) [Barlevy and Neal,](#page-47-3) [2012;](#page-47-3) [Loyalka](#page-51-2) *et al.*, [2019\)](#page-51-2).[3](#page-2-1) In addition, empirical evidence from a recent meta-analysis suggests teacher incentive schemes in the US that featured rank-ordered tournaments outperform other designs [\(Pham](#page-53-0) *et al.*, [2021\)](#page-53-0). Rank-order tournaments are also harder to game, and since the rewards are based on ordinal measures, they can more easily accommodate changes in exam formats and other system-wide changes [\(Neal,](#page-52-1) [2011\)](#page-52-1). Yet, the theoretical advantages of tournaments may not materialise in practice if teachers find it difficult to determine how to react to such schemes [\(Charness and Kuhn,](#page-48-7) [2011\)](#page-48-7).
We conducted a randomised experiment to examine the effectiveness (and cost-effectiveness) of two individual-level teacher incentive schemes in a nationally representative set of 180 Tanzanian public schools. We randomly assigned 60 schools to a "Levels" scheme with multiple proficiency thresholds
2 In health-care settings, performance-based incentives with clear targets have been shown to have a positive effect on performance partly because these targets clarify what health workers' responsibilities and tasks are [\(Miller and Babiarz,](#page-52-2) [2014;](#page-52-2) [Renmans](#page-53-1) *et al.*, [2016\)](#page-53-1).
3The conditions under which Pay for Percentile, the rank-order tournaments we study, yields socially optimal levels of effort are outlined by [Barlevy and Neal](#page-47-3) [\(2012\)](#page-47-3). A key assumption is that the social planner maximises total returns from learning minus total cost. Another critical assumption is that the production function of human capital is linear in teacher effort and separable between the student's initial learning levels and other factors affecting learning.
that correspond to important curricular milestones.[4](#page-3-0) We randomly assigned 60 schools to a "Pay for Percentile" (a rank-order tournament) scheme based on [Barlevy and Neal](#page-47-3) [\(2012\)](#page-47-3). The per-student bonus budget was equalised (ex-ante) across grades, subjects, and treatment arms to facilitate comparisons. The average teacher bonus was approximately 3.5% of the annual net salary (roughly half a month's pay).[5](#page-3-1) We randomly assigned 60 schools to a control group. In all three groups, teachers were provided with baseline student reports so they were aware of each student's initial skill (or proficiency) level. Our main outcome is student performance on externally administered tests in math, Kiswahili, and English in first, second, and third grades.[6](#page-3-2) Following [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3), we evaluate the incentive programs using data from both the incentivised (or "high-stakes") test administered to all students to determine teacher bonuses and a non-incentivised (or "low-stakes") test administered to a sample of students for research purposes.[7](#page-3-3)
In the 60 schools assigned to receive incentives based on proficiency targets (the "Levels" arm), teachers earned bonuses based on their students' mastery of several grade-specific skills. We included several thresholds to mitigate concerns that incentive programs using single-proficiency thresholds encourage teachers to focus on students close to the passing threshold. The skills thresholds were salient milestones based on the national curriculum, ranging from basic (e.g., number recognition) to more complex skills (e.g., multiplication) to allow teachers to earn rewards from a wide range of students. As reward payments for each skill were inversely proportional to the number of students that passed the skill, harder-to-pass skills were rewarded more.[8](#page-3-4)
In the 60 schools assigned to the Pay for Percentile arm, students were first tested and assigned to one of several "baseline ability groups" based on their test scores. Teachers' rewards were proportional to their students' rankings within each group. The system does not penalise teachers who serve disadvantaged students because it explicitly accounts for the differences in initial student performance across teachers.
4To the best of our knowledge, this is the first documented implementation of proficiency-based teacher incentives with multiple (curricular-based) thresholds.
5Similar incentive sizes were used in [Fryer](#page-49-3) [\(2013\)](#page-49-3); [Glewwe](#page-49-4) *et al.* [\(2010\)](#page-49-4); [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3); [Muralidharan and Sundararaman](#page-52-3) [\(2011\)](#page-52-3); [Lavy](#page-50-3) [\(2002\)](#page-50-3); [Ladd](#page-50-4) [\(1999\)](#page-50-4); [Vigdor](#page-53-2) [\(2008\)](#page-53-2). See [Leigh](#page-51-4) [\(2012\)](#page-51-4) for details.
6English was dropped from the national curriculum in first and second grades during the experiment. Therefore, we focus on math and Kiswahili test scores. The analysis of English scores is in the Appendix.
7Both types of tests were conducted in control schools. However, the "incentivised" test results did not trigger payments in these schools.
8Since payments were determined ex-post based on pass rates, teachers faced uncertainty about the exact bonus sizes. However, an individual teacher's effort has a negligible effect on the aggregate pass rate and teachers likely have sufficient ex-ante information (e.g., through experience) to have reasonable predictions about the pass rates. We formalise this intuition in Appendix [D.](#page-95-0)
Given the well-documented concerns about teachers misunderstanding incentive designs [\(Goodman and](#page-50-5) [Turner,](#page-50-5) [2013;](#page-50-5) [Fryer,](#page-49-3) [2013\)](#page-49-3), we developed information packets that used culturally appropriate scripts and examples, and budgeted extra time to explain the design details.
We report two main findings. First, both incentives schemes improve learning outcomes compared to the control group, especially when we examine the results from the incentivised tests. Focusing on the results at the end of the second year of the program, the composite test scores of students in the Levels treatment were 0.22*σ* higher (p-value < 0.01) compared to the control. For students in the Pay for Percentile treatment, composite test scores were 0.13*σ* higher (p-value 0.027) compared to the control. For both treatments, gains were lower on the non-incentivised tests, but the magnitudes remained meaningful. For non-incentivised tests, composite tests scores were 0.095*σ* (p-value 0.036) and 0.041*σ* (p-value 0.33) higher in Levels schools and Pay for Percentile schools, respectively, when compared to control schools.[9](#page-4-0) These learning gains in incentivised subjects were not at the expense of learning in other subjects: we do not find any evidence of negative treatment effects on science (which was non-incentivised).
Second, despite the theoretical predictions and empirical evidence on the effectiveness of tournament schemes relative to alternative designs, we find that composite test scores for the Levels incentive system increased (at least) as much as those in the Pay for Percentile system. At the end of the second year, the estimated treatment effect on the incentivised composite test score in Levels schools was 0.096*σ* higher (pvalue 0.097) than the estimates for Pay for Percentile schools. The treatment effect on the non-incentivised composite test score shows a similar pattern, although the difference is smaller (0.053*σ*) and statistically insignificant (p-value 0.27).[10](#page-4-1) Factoring in the administrative and implementation costs, our analysis shows that the Levels scheme is more cost-effective than the Pay for Percentile scheme. Although theory suggests that Pay for Percentile might produce more equitable learning gains relative to the Levels system, we find similar learning gains in the second year across all five quintiles of the student baseline test score
9As the test content was similar across tests, the differences in treatment effects are likely due to differences in student test-taking effort [\(Levitt](#page-51-5) *et al.*, [2016;](#page-51-5) [Gneezy](#page-49-5) *et al.*, [2019\)](#page-49-5). See Section [3.2](#page-18-0) and Appendix [E](#page-106-0) for details on the design and implementation of both tests.
10Year 2 results are arguably more informative because teachers learn how to better respond to the incentives over time.
distribution (using composite test scores) in both treatment arms. This suggests that multiple thresholds can mitigate the inequality concerns associated with proficiency systems.[11](#page-5-0)
We use a comprehensive set of survey data from school administrators, teachers, and students and data from classroom observations to shed light on theoretically relevant mechanisms. Because measuring teacher effort and behaviour using these methods is challenging due to Hawthorne and/or John Henry effects, as well as social desirability bias [\(Muralidharan and Sundararaman,](#page-52-4) [2010;](#page-52-4) [Muralidharan,](#page-52-5) [2017\)](#page-52-5), we try to mitigate these concerns using additional measures such as external (to the classroom) observations of teacher behaviour. In addition, our enumerators examine a sample of student notebooks to document teacher feedback to students. Although we find no changes in teacher absenteeism in either treatment, data from the external classroom observations suggest that teachers in both treatment groups increased teaching time and reduced time off-task, although these effects were imprecisely estimated. Data from student self-reports suggest that the classroom dynamics were more conducive to learning. Teachers in both groups were more likely to call students by their names and were less likely to use corporal punishment, although these estimates were also noisy.[12](#page-5-1) We also find that teachers in both systems understood the incentive designs. The high levels of teacher comprehension were partly due to our implementation and communication efforts. Further, teachers in both treatments had high expectations about their earning potential. Teachers expected to earn almost twice the actual average payment in both groups. However, teachers in the Pay for Percentile schools reported they expected to receive 18% lower bonus payments, on average, compared to their Levels counterparts. To the extent that expectations mirror effort, these patterns in expectations may also partly explain the increases in learning outcomes in both treatment groups and the instances where we find smaller treatment effects in the Pay for Percentile treatment relative to the Levels treatment.
11Theory predicts that if the productivity of teacher effort is constant across initial student learning levels, teachers will focus less on students in the tails of the ability distribution in proficiency systems. In contrast, they would focus on all students under a Pay for Percentile system [\(Barlevy and Neal,](#page-47-3) [2012\)](#page-47-3). However, this difference between the designs becomes less pronounced when the productivity of teacher effort increases with students' initial learning levels. See Appendix [D](#page-95-0) for more details.
12[Muralidharan](#page-52-5) [\(2017\)](#page-52-5) discusses the challenges of measuring teacher effort in the field, especially with limited research budgets. Many articles studying teacher incentives do not find any measurable response in teacher effort, even when they find treatment effects in student learning (e.g., [Muralidharan and Sundararaman](#page-52-3) [\(2011\)](#page-52-3); [Loyalka](#page-51-2) *et al.* [\(2019\)](#page-51-2)). This is perhaps unsurprising given that teachers can adjust effort on many margins that are difficult to measure (e.g., pedagogy, time-on-task, homework, socio-emotional support) in response to incentives linked to student learning. Advances and cost reductions in technology have made video recordings of classes more feasible. See [Brown and Andrabi](#page-48-8) [\(2020\)](#page-48-8) for an example.
Our study contributes to the debate about how to best structure and design teacher incentives. [Breeding](#page-48-4) *[et al.](#page-48-4)* [\(2021\)](#page-48-4) find that most teacher incentive schemes yield little to no student learning improvements. Further, they suggest these mediocre results are often the consequence of design choices that mute the incentives. However, there is a limited set of adequately powered experimental studies comparing different teacher incentive designs. These comparisons include individual versus group incentives [\(Muralidharan](#page-52-3) [and Sundararaman,](#page-52-3) [2011\)](#page-52-3); bonuses based on average class test scores, value-added, and Pay for Percentile, all under a common rank-order tournament structure [\(Loyalka](#page-51-2) *et al.*, [2019\)](#page-51-2); in-kind rewards versus public recognition [\(Barrera-Osorio](#page-47-4) *et al.*, [2022\)](#page-47-4); and bonuses that featured a loss-aversion framing compared to a traditional bonus scheme [\(Fryer](#page-49-6) *et al.*, [Forthcoming\)](#page-49-6).[13](#page-6-0) Generally, previous studies focused on comparisons featuring more intricate and (theoretically) effective incentives. In contrast, we compare a modified version of a commonly used proficiency design (Levels) that is theoretically less effective and a system generally considered to encourage much greater and potentially socially optimal levels of effort (Pay for Percentile). Contrary to the theoretical predictions, a multiple threshold system tied to foundational literacy and numeracy objectives (Levels) is more cost-effective at improving learning outcomes for all students in early grades compared to a more sophisticated, cost-equivalent (in terms of bonuses) rank-order tournament (or Pay for Percentile) scheme.[14](#page-6-1) Our results reinforce the importance of testing theoretical predictions in real-world settings as practical constraints and other unforeseen circumstances can cause individuals to deviate from their predicted behaviour. They also highlight the importance of the practical limitations of tournaments outlined in [Charness and Kuhn](#page-48-7) [\(2011\)](#page-48-7) and shed light on the trade-offs faced by education authorities who have to consider the (cost-) effectiveness and feasibility of implementing different teacher incentive designs, often with limited information about the education production function.[15](#page-6-2)
13A related literature compares the effectiveness of different incentive designs for healthcare providers [\(Singh and Masters,](#page-53-3) [2018;](#page-53-3) [Mohanan](#page-52-6) *et al.*, [Forthcoming\)](#page-52-6).
14We are only aware of four studies in developing country contexts that specifically evaluate Pay for Percentile schemes. [Loyalka](#page-51-2) *[et al.](#page-51-2)* [\(2019\)](#page-51-2) find that Pay for Percentile incentives increased test scores among math teachers in Chinese schools and that is more effective than rank-order tournaments based on test score levels or value-added measures — this is the only study we are aware of that compares Pay for Percentile to alternative incentive structures. [Gilligan](#page-49-7) *et al.* [\(2019\)](#page-49-7) find that Pay for Percentile has no impact on student learning in Ugandan schools, except for top students in schools with textbooks. [Leaver](#page-51-6) *et al.* [\(2021\)](#page-51-6) study the extensive (recruitment) and intensive (effort) effect of Pay for Percentile in the context of Rwandan primary teachers. On the intensive margin (the margin we study), Pay for Percentile increases teacher effort and improves student learning outcomes. [Brown and Andrabi](#page-48-8) [\(2020\)](#page-48-8) find that pay for percentile induces positive sorting of teachers in a network of private schools in Pakistan. Finally, we are only aware of one study in a developed country studying the impact of Pay for Percentile [\(Fryer](#page-49-6) *et al.*, [Forthcoming\)](#page-49-6).
15Our time frame does not allow us to examine how teachers and administrators would respond when they gained more experience with the incentive schemes (e.g., if the incentive schemes were permanently adopted by the government.)
# **2 Experimental Design**
# **2.1 Context**
Tanzania allocates about one-fifth of overall government spending (roughly 3.5% of GDP) to education [\(World Bank,](#page-54-2) [2017b\)](#page-54-2). Much of this spending has been devoted to promoting educational access. As a consequence, net enrolment rates in primary school increased from 53% in 2000 to 80% in 2014 [\(World](#page-54-2) [Bank,](#page-54-2) [2017b\)](#page-54-2). Despite these gains in educational access, educational quality remains a major concern. Resources and materials are scarce. For example, in 2017 only 14% of schools had access to electricity and just over 40% had access to potable water [\(World Bank,](#page-54-2) [2017b\)](#page-54-2). Nationwide, there are approximately 43 pupils per teacher [\(World Bank,](#page-54-2) [2017b\)](#page-54-2), although early grades often have much larger class sizes. In 2013, approximately five pupils shared a single mathematics textbook, while 2.5 pupils shared a reading textbook [\(World Bank,](#page-54-2) [2017b\)](#page-54-2). Student learning levels are also low. In 2012, data from nationwide assessments showed that only 38% of children aged 9-13 could read and do arithmetic at the grade 2 level, suggesting that educational quality is a pressing policy problem [\(Uwezo,](#page-53-4) [2013\)](#page-53-4).
Limited accountability within the education system is one driver of poor educational quality. Quality assurance systems (e.g., school inspectors) typically focus on superficial issues, rather than issues that may affect learning [\(Mbiti,](#page-51-0) [2016\)](#page-51-0). Teacher absence rates further reflect the accountability vacuum. Data from unannounced spot checks shows that 14% of teachers were absent from school and only 47% of the teachers at the school were in the classroom [\(World Bank,](#page-54-3) [2015\)](#page-54-3). Adding up school absence, classroom absence, and time spent on non-teaching activities, almost 50% of planned instructional time is lost each day [\(World Bank,](#page-54-3) [2015\)](#page-54-3).
To address education quality concerns, Tanzanian teachers' unions have been actively lobbying for better pay. Yet, the correlation between teacher compensation and student learning is extremely low [\(Kane](#page-50-6) *et al.*, [2008;](#page-50-6) [Bettinger and Long,](#page-47-5) [2010;](#page-47-5) [Woessmann,](#page-54-4) [2011;](#page-54-4) [de Ree](#page-49-8) *et al.*, [2018\)](#page-49-8). Moreover, teacher salaries in 2016 were relatively high — approximately 500,000 TZS per month (∼ US\$250) or over 3 times GDP per capita [\(World Bank,](#page-54-2) [2017b\)](#page-54-2) — comprising about 60% of the education budget.[16](#page-8-0) Despite Tanzanian teachers' relatively attractive wages, the teachers' union called a strike in 2012 to demand a 100% increase in pay [\(Reuters,](#page-53-5) [2012;](#page-53-5) [PRI,](#page-53-6) [2013\)](#page-53-6).[17](#page-8-1)
# **2.2 Interventions and Implementation**
The interventions in this study were developed in close collaboration with Twaweza, an East African civil society organisation that focuses on citizen agency and public service delivery. The interventions were part of a series of projects launched under a broader program umbrella known as KiuFunza ('Thirst for learning' in Kiswahili).[18](#page-8-2)
The KiuFunza program targets teachers in grades 1, 2, and 3 who are responsible for teaching Kiswahili, English, and math (arithmetic). A budget of US\$150,000 per year for teacher and headteacher incentives was split between the two treatment arms in proportion to the number of students enrolled. As a result, the prize money in each treatment arm was approximately US\$3 per student. In partnership with EDI (a Tanzanian research firm), Twaweza and a set of local district partners implemented all interventions. Headteachers were offered a bonus of 20% of the combined bonus of all incentivised teachers in their school.[19](#page-8-3)
Within each intervention arm, Twaweza distributed information describing the program in early 2015 and 2016: first to teachers and headteachers, and then to their respective communities via public meetings. From the program's onset, Twaweza informed teachers the program would last two years. The implementation teams also conducted mid-year school visits to re-familiarise teachers with the program, gauge teacher understanding of the bonus payment mechanisms, and answer any remaining questions.
At the end of the school year, all students in grades 1, 2, and 3 in every school, including control schools, were tested in Kiswahili, English, and math. Because this test was used to determine teacher incentive
16The average teacher in a sub-Saharan African country earns almost four times GDP per capita, compared to OECD teachers who earn 1.3 times GDP per capita [\(OECD,](#page-53-7) [2017;](#page-53-7) [World Bank,](#page-54-2) [2017b\)](#page-54-2).
17In recent years, other teacher strikes to demand pay increases have occurred in South Africa, Kenya, Guinea, Malawi, Swaziland, Uganda, Benin, and Ghana.
18The first set of interventions under this program were launched in 2013, lasted until 2014, and were evaluated by [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3).
19Twaweza included headteachers in the incentive design to ensure that they would be stakeholders in improving learning outcomes. Likewise, any scaled-up teacher incentive program would feature bonuses for headteachers.
payments, it was considered "high-stakes" (from the teachers' perspective). Our non-incentivised research test was conducted on a different day but within a few weeks from the incentivised test. Both sets of tests were based on the Tanzanian curriculum. They were developed by Tanzanian education professionals, using formats of the Uwezo learning assessment framework.[20](#page-9-0) We provide additional details about the design and implementation of both types of tests in Appendix [E.](#page-106-0)
## **2.2.1 Pay for Percentile design**
The Pay for Percentile design used in our intervention is based on research by [Barlevy and Neal](#page-47-3) [\(2012\)](#page-47-3). They show that this incentive structure can, under certain conditions, induce teachers to exert socially optimal levels of effort. A necessary condition for Pay for Percentile to induce optimal effort is that teachers believe they compete in properly seeded (or fair) contests. To achieve this, the Pay for Percentile scheme uses a modified rank-order tournament structure that accounts for the heterogeneity in students' baseline learning levels across classrooms (and teachers). Specifically, the system divides students into groups based on their academic achievement (or "ability"), and a separate rank-order tournament is conducted for each group. Teachers are then rewarded based on their students' rank order within each ability group. Without this adjustment, teachers in schools that served students from affluent backgrounds would be advantaged, and those serving less-affluent students may be discouraged from exerting effort.
To implement this system in practice, we created student groups with similar initial learning levels based on test score data from the previous school year for each subject-grade combination (see Appendix [C.1](#page-90-0) for details on the number and size of the groups). Students without test scores in second and third grade were grouped in an "unknown" ability group.[21](#page-9-1) Since none of the first-grade students had incoming test scores, we created broad country-level ability groups and assigned all first-grade students within a school to the same group based on the school's historical average test scores. Thus, all first-grade students within a school were assigned to the same group.
20Uwezo learning assessments have been routinely conducted in Kenya, Tanzania, and Uganda since 2010.
21Roughly 20% of students are grouped into the "unknown" ability group. This includes newly enrolled students and students who were enrolled but were not tested at baseline for some reason.
To compute the payment structure, we divide the total prize money in this treatment arm equally across grades and subjects. We then apportion the subject-grade budget to each ability group in proportion to the total number of students in the grade who are in each ability group. At the end of the year, we ranked students within each ability group according to their endline test scores. Within each ability group, we assigned teachers points proportional to the rank of their students. For a given ability group, a teacher would receive 99 points for a student in the top 1% of the group and zero points for a student in the bottom 1% of the group. In other words, the rewards increase linearly in rank. The total amount of money paid per point is the same across all groups in all subjects and all grades. Students without an endline test score were given a zero score and were ranked at the bottom of their ability group. Thus, there were no incentives to exclude academically weaker students.
For example, suppose there is a total of US\$1,000 for teacher incentives and that there are two ability groups with 40 and 60 students. Accordingly, the total budget for teacher bonuses in each ability group would be US\$400 and US\$600. In each ability group, the total bonus would be equal to the sum of all teacher rewards or
$$X = \sum_{i=1}^{100} b * (i-1) * \frac{N}{100}$$
(1)
where X is the total budget for teacher bonuses in each ability group, N is the number of students in each ability group, i indexes a student's percentile rank on the endline test, and i is the teacher reward per point. Since $\sum_{i=1}^{100} (i-1) = 4,950$ , the reward per point (i) is roughly i0.20 for both groups. Thus, in this example, if a student was in the top 1% of their ability group, their teacher would earn i0.2 or US\$19.8. Conversely, a median student would earn their teacher i0.2 or US\$10. In the first year of our study, the total bonus available to teachers in Pay for Percentile schools was US\$70,820 and total enrolment was 22,296. For each grade and subject, teachers earned US\$1.77 for each student in the top 1% and US\$0.89 for each student in the 50th percentile.
Although this design can deliver socially optimal levels of effort under certain conditions, it may be challenging to implement at scale, particularly in settings with weak administrative capacity, such as Tanzania. For instance, maintaining child-level panel databases is a non-trivial administrative challenge.
Moreover, teachers may find the Pay for Percentile system difficult to grasp. It presents each teacher with a series of tournaments (for each ability group in each subject that they teach); therefore, the bonus payoff is relatively hard to predict, even if the design guarantees a fair system. Furthermore, competing against teachers from schools across the country introduces uncertainty that may dilute the incentive.
# **2.2.2 Proficiency thresholds (Levels) design**
Proficiency-based systems are easier for teachers to understand and provide more actionable targets than rank-order or value-added tournaments. Consequently, such systems are likely to increase motivation among teachers and headteachers; however, they have well-known limitations. For example, they cannot adequately account for differences in the initial distribution of student preparation across schools and classrooms. Moreover, this type of system can encourage teachers to focus on students close to the proficiency threshold, at the expense of students who are well above or below the threshold [\(Neal and](#page-52-7) [Schanzenbach,](#page-52-7) [2010\)](#page-52-7). To mitigate this concern, our Levels design features multiple thresholds ranging from basic skills to more advanced ones in the curriculum. This design allows teachers to earn bonuses for helping a broader set of students, including students with lower and higher baseline test scores.[22](#page-11-0) [Miller](#page-52-2) [and Babiarz](#page-52-2) [\(2014\)](#page-52-2) argue that incentive designs based on "bright-line" performance thresholds (and goals) can be effective in helping service providers — in this case, teachers — to focus on achieving these goals. They also argue that bright-line designs are well suited to helping providers focus on achieving important outcomes.[23](#page-11-1)
In Levels schools, teachers are paid in proportion to the number of skills that their students in grades 1-3 master in mathematics, Kiswahili, and English at the end of the school year. The total budget is split across grades in proportion to the number of students enrolled in each grade. The budget is then divided equally among subjects and skills within each subject. The bonus per pass for each skill equals the skill budget divided by the number of students passing the skill. For example, suppose the budget allocated
22As discussed in Appendix [D,](#page-95-0) a key practical challenge is ensuring that the thresholds are sufficiently close together to prevent teachers from ignoring students who fall between two thresholds. Appendix [C.2](#page-94-0) shows the passing thresholds are indeed spread across the ability distribution.
23In the health sector, [Miller and Babiarz](#page-52-2) [\(2014\)](#page-52-2) argue bright-lines may be especially appropriate when thresholds have clinical significance (e.g., vaccination rates). In our early grade education setting, the fundamental nature of the numeracy and literacy thresholds in our design corresponds with these criteria.
to one grade for demonstrating proficiency in addition (a math skill) is US\$1,000. If there are 500 students in the grade, and 250 pass the addition portion of the math test, then a teacher would receive US\$4 per pass, that is, for every student in her class that was proficient in addition.
Table [1](#page-12-0) shows the skills (i.e., the thresholds) tested in each grade-subject combination and the corresponding (ex-post) payment per student that each teacher would receive. Since the per-pass bonus paid ex-post is equal to the skill budget divided by the number of students passing the skill, the budget for easier-to-obtain skills is spread across more students — resulting in a lower per-pass bonus. Conversely, harder-to-obtain skills have a higher per-pass bonus. Thus, teachers have the potential to earn larger bonuses if their students are proficient in a larger number of skills, especially harder-to-obtain skills.[24](#page-12-1)
Table 1: Skills tested in the Levels schools
| Kiswahili | Math | | | | |
|----------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--|--|--|--|
| Grade 1 | | | | | |
| Letters (TZS 1,992 or ∼US\$ 0.95)
Words (TZS 1,619 or ∼US\$ 0.77)
Sentences (TZS 2,057 or ∼US\$ 0.98) | Counting (TZS 513 or ∼US\$ 0.24)
Numbers (TZS 750 or ∼US\$ 0.36)
Inequalities (TZS 649 or ∼US\$ 0.31)
Addition (TZS 748 or ∼US\$ 0.36)
Subtraction (TZS 821 or ∼US\$ 0.39) | | | | |
| Grade 2 | | | | | |
| Words (TZS 1,192 or ∼US\$ 0.57)
Sentences (TZS 1,297 or ∼US\$ 0.62)
Paragraphs (TZS 2,214 or ∼US\$ 1.05) | Inequalities (TZS 803 or ∼US\$ 0.38)
Addition (TZS 1,136 or ∼US\$ 0.54)
Subtraction (TZS 1,374 or ∼US\$ 0.65)
Multiplication (TZS 1,732 or ∼US\$ 0.82) | | | | |
| Grade 3 | | | | | |
| Story (TZS 1,709 or ∼US\$ 0.81)
Comprehension (TZS 1,530 or ∼US\$ 0.73) | Addition (TZS 694 or ∼US\$ 0.33)
Subtraction (TZS 900 or ∼US\$ 0.43)
Multiplication (TZS 3,660 or ∼US\$ 1.74)
Division (TZS 1,820 or ∼US\$ 0.86) | | | | |
Note: This table shows the skills tested in each subject and grade. In parentheses are teachers' payments for each student who masters each skill in the first year.
24Enrolment at each school is on average 1.6% of total enrolment across Levels schools. Thus, the total number of a teacher's students passing the threshold has a negligible effect on the overall pass rate across schools. Hence, we can rule out teachers strategically choosing how many students to push over a threshold to maximise earnings.
# **2.2.3 Teacher understanding of the incentive designs**
Teacher incentive programs may be ineffective if teachers cannot understand the program details and, therefore, do not optimally allocate their effort [\(Goodman and Turner,](#page-50-5) [2013;](#page-50-5) [Loyalka](#page-51-2) *et al.*, [2019\)](#page-51-2). These concerns are potentially more important in developing country contexts where public institutions may be less able to disseminate the details of an incentive program to teachers effectively.
During baseline and midline school visits, teams reinforced teachers' familiarity with the programs' main features. We developed culturally appropriate materials to enhance teachers' understanding of the incentive schemes, including Q&A formats, examples, and illustrations. For example, in Pay for Percentile schools, we explained that students would be grouped into separate contests based on their initial abilities, ensuring that each contest would be fair. To make our explanation clear, we used an analogy of a footrace. We explained that a race featuring one fast runner competing against slower opponents would be unfair. A fairer system would group runners into separate races based on their speed.[25](#page-13-0)
During our visits, we tested teachers to ensure they understood the details of the incentive program they were assigned to. We then conducted a review session to discuss the answers to the test questions to ensure that teachers understood the design details. Because we asked different questions about each incentive scheme during each survey round (baseline, midline, and endline), we cannot compare the trends in understanding over time or across treatments. However, despite the lack of comparability, teacher comprehension was generally high and roughly equal across both types of incentive programs. For example, at the end of the second year, 70% of teachers in Levels schools knew that the amount of money paid per skill obtained by their students depended on the total number of students that passed across Tanzania. Over 90% of teachers in Pay for Percentile schools were aware that a student from a low ability group ranked at the top of his group at the end of the year would give them a larger bonus than a student in the highest ability group ranked low among their peers.
25We worked closely with Twaweza's communications unit to develop our dissemination strategy and communications. The communications unit is experienced and highly specialised in developing materials to inform and educate the general public in Tanzania. Appendix [F](#page-113-0) provides a copy of the material used to explain the interventions.
# **2.3 Conceptual Framework and Theoretical Predictions**
We develop a simple model of student learning to highlight the differences in teacher effort under the two incentive schemes. For brevity, we first outline the main features of the model and then discuss the predictions. We discuss the details of the model in Appendix [D.](#page-95-0)
We assume students' learning depends on their baseline learning level and their teacher's effort, which is costly to exert, and an idiosyncratic random shock. Individual students have different baseline learning levels, and all teachers are assumed to be equally skilled. Because our study compares the performance of the two incentive systems, in the model we assume teachers only respond to extrinsic (monetary) rewards for tractability. We impose three parametric assumptions to obtain sharper predictions about teacher effort under each incentive scheme. First, we assume that the cost of teacher effort is a quadratic function. Second, we assume that student baseline learning levels are uniformly distributed and verify that we obtain similar predictions if we were to impose a normal distribution instead. Finally, we assume that idiosyncratic random shocks to student learning are normally distributed. We then use simulations to study teachers' utility-maximising efforts under the different incentive schemes. In our framework, the utility-maximising effort is achieved when the marginal benefit (in terms of extra monetary rewards) equals the marginal cost of effort. We also study how teachers direct their effort toward different types of students, a process also referred to as "triage" [\(Loyalka](#page-51-2) *et al.*, [2019;](#page-51-2) [Gilligan](#page-49-7) *et al.*, [2019\)](#page-49-7).
We obtain two main predictions. First, our simulations predict that total teacher effort (and hence student learning) will be greater under Pay for Percentile than Levels. This result is partly driven by the fact that under the Levels system student baseline levels will affect teacher rewards, while they will not play an important role in the Pay for Percentile system if students are grouped into the separate rank-order tournaments appropriately. Both systems are predicted to improve teacher effort and student learning relative to the status quo in control schools if the rewards are sufficiently large to compensate for the costs of effort. This is consistent with the theoretical results of [Barlevy and Neal](#page-47-3) [\(2012\)](#page-47-3) and the empirical results of [Loyalka](#page-51-2) *et al.* [\(2019\)](#page-51-2).
Second, teacher "triage" depends on the correlation between the productivity of teacher effort and student baseline ability. Suppose the productivity of teacher effort is constant for all types of students. In that case, teachers under a Pay for Percentile scheme will (optimally) exert equal effort across the entire distribution of students. By contrast, teachers under the Levels system would (optimally) focus on students in the middle of the distribution and exert little effort at the tails. However, if (for example) the productivity of teacher effort is positively correlated with student baseline levels, then under Pay for Percentile teachers would optimally exert more effort towards better-prepared students. Under Levels, teachers would continue to optimally exert more effort towards students in the middle of the distribution and less effort at the tails.
Despite these predictions, there are several considerations that may influence how teachers respond to the incentives in practice. First, the monetary benefits must be sufficiently high to elicit behavioural responses from teachers. Second, both systems have to be appropriately designed. For instance, the groups (or contests) in the Pay for Percentile system must consist of students of similar ability, and the thresholds in the Levels systems cannot be too "far apart" or else teachers might ignore students who are in between these thresholds. Third, the incentives will be less effective if teachers cannot understand the incentives or the cognitive costs of understanding the rules are too high. Fourth, the incentives will also be less effective if teachers do not trust the incentive scheme. More transparent systems can potentially improve teacher understanding, their beliefs about the relationship between effort and rewards, and perhaps engender more trust in the scheme. Thus, although theory predicts that Pay for Percentile will improve learning more than Levels, the greater transparency of Levels may improve teachers' relative response to that system relative to Pay for Percentile. Further, teachers under Levels might be better able to respond due to the alignment of the Levels thresholds with important curriculum milestones.
# **2.4 A Note on Curriculum Reform and English Language Teaching**
As Kiswahili is the official language of instruction in primary schools in Tanzania, English is taught as a second language. However, English is rarely spoken outside of the classroom, so English language skills are quite low in Tanzania. For instance, only 12% of grade 3 students were proficient at the grade 2 level in English [\(Uwezo,](#page-53-8) [2012\)](#page-53-8). Given the challenges of teaching English in Tanzania, the subject was removed from the national curriculum in grades 1 and 2 in 2015 to allow teachers to focus on numeracy and literacy in Kiswahili. English was still taught in grade 3, under a revised curriculum. However, the Education Ministry provided little guidance on how to transition to the new curriculum and, as a result, there was substantial variation in its implementation. Some schools stopped teaching English in 2015, while others continued until 2016. In addition, there was no official guidance on whether to use grade 1 English materials in grade 3, as no new books were issued that reflected the curriculum changes. To maintain consistency between the curriculum and KiuFunza incentives, Twaweza dropped English from the incentives in grades 1 and 2 in 2016 but included grade 3 English teachers. To avoid confusion, we also communicated that our end-of-year English test in 2016 would still use the pre-reform grade 3 curriculum. Given these issues in the curriculum reform's implementation, it is unclear how to interpret the results for English. In addition, these estimates are less policy-relevant after the reform. Therefore, we only present mathematics and Kiswahili results in the main text to facilitate the analysis. In addition to this change, the curriculum reform prescribed that multiplication would be dropped from the grade 2 curriculum.
# **3 Data and Empirical Specification**
# **3.1 Sample Selection**
We evaluated the teacher incentive programs using a randomised design. The study was carried out in a nationally representative set of 180 Tanzanian public schools. These schools were part of a previous field experiment — studied by [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3) — where all students in grades 1, 2, and 3 were tested at the end of 2014. These tests provided the baseline student-level test score information required to implement the Pay for Percentile treatment.[26](#page-16-0) As mentioned above, a necessary condition for the Pay for Percentile to
26These tests were administered before schools were assigned to the treatment groups discussed in this paper. Online appendices [B](#page-88-0) and [E](#page-106-0) provide more details.
deliver optimal levels of effort is that teachers believe they compete in fair contests. Thus, having reliable information about students' initial learning levels was key.[27](#page-17-0)
The 180 schools in the sample are distributed across ten districts. The districts were randomly selected to participate in the experiment (see Figure [1\)](#page-17-1), which provides external validity to our results across Tanzania [\(Muralidharan and Niehaus,](#page-52-8) [2017\)](#page-52-8).[28](#page-17-2) Within each district, we randomly allocated schools to one of our three experimental groups. Thus, six schools were assigned to the Levels treatment in each district, six schools to the Pay for Percentile treatment, and six schools served as controls. In total, there were 60 schools in each group. The treatment assignment was also stratified by treatment of the previous field experiment and by an index of the overall learning level of students in each school. Further details are provided in Appendix [B.](#page-88-0)
Figure 1: Districts in Tanzania from which schools are selected
*Note: We drew a nationally representative sample of 180 schools from a random sample of 10 districts in Tanzania (shaded).*
27We do not have data on whether teachers believe they are competing in a fair contest. However, before receiving any payment, over 90% of teachers agreed or strongly agreed that the amount paid by Twaweza would be fair, suggesting teachers think the contests are fair.
28The program was implemented in 11 districts, as one district was included non-randomly by Twaweza for piloting and training. We did not survey schools in the pilot district.
# **3.2 Data and Balance**
Over the two-year evaluation, our survey teams visited each school at the beginning and end of the school year. We gathered detailed information about each school from the headteacher, including facilities, management practices, and headteacher characteristics. We also conducted individual surveys with the teachers in our evaluation to determine personal characteristics, including education and experience, and effort measures, such as teaching practices and teacher absence. In addition, we conducted two types of classroom observations, in which we recorded teacher-student interactions.
Within each school, we surveyed and tested a random sample of 40 students (10 students from grades 1, 2, 3, and 4). Grade 4 students were included in our research sample to measure potential spillovers to other grades. Students in grades 1, 2, and 3 who were sampled in the first year of the program were tracked over the two-year evaluation period. Due to budget constraints, students in grade 4 in the first year were not tracked into grade 5. In the second year of the program, we sampled an additional 10 incoming Grade 1 students. We collected a variety of data from our student sample, including test scores, individual characteristics, such as age and gender, and perceptions of the school environment. Crucially, the test scores collected on the sample of students are "low-stakes" for teachers and students. We supplemented the results from this set of non-incentivised student tests with the results from the incentivised tests used to determine teacher bonus payments and conducted in all schools, including control schools. Most articles studying teacher performance pay use incentivised tests to measure the overall treatment effects. However, it is unclear whether incentivised or non-incentivised tests are better for measuring treatment effects. We, therefore, present results from both tests for completeness.[29](#page-18-1)
The incentivised and the non-incentivised tests covered very similar content (subject order, question type, phrasing, and difficulty level). The non-incentivised test had more questions for each subject to avoid bottom- and top-coding and included an "other subject" module at the end to test spillover effects. Further, even though both tests were administered individually to students, the testing environment was
29As argued by [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3): "The confirmation that test-taking effort is a salient component of measured test scores by [Levitt](#page-51-5) *[et al.](#page-51-5)* [\(2016\)](#page-51-5) and [Gneezy](#page-49-5) *et al.* [\(2019\)](#page-49-5) presents a conundrum for education researchers as to what the appropriate measure of human capital should be for assessing the impact of education interventions. On one hand, low-stakes tests may provide a better estimate of a true measure of human capital that does not depend on external stimuli for performance. On the other hand, test-taking effort is costly, and students may not demonstrate their true potential under low-stakes testing, in which case, an 'incentivised' testing procedure may be a better measure of true human capital."
different. Non-incentivised tests were administered during a regular school day by survey enumerators. In contrast, the incentivised test was more "official" as all students in grades 1-3 were tested on a specified day. On the test day, a Twaweza test team would administer the tests in dedicated classrooms, with headteachers and teachers managing the flow of students. In addition, most schools used the incentivised test as the official end-of-year test. Several measures were introduced to enhance test security. First, to prevent test-taking by non-target grade candidates, students could only be tested if their name had been listed and their photo was taken at baseline. Second, each student was assigned one test randomly selected out of ten test versions to prevent copying during the test. Finally, Twaweza teams handled, administered, and electronically scored tests without teacher involvement. Section [E](#page-106-0) provides more details on the design and implementation of both tests.
Most student, school, teacher, and household characteristics are balanced across treatment arms (See Table [2,](#page-20-0) Column 4). The average student in our sample was 8.9 years old in 2013, went to a school with 679 students, and was taught by a teacher who was 38 years old. In addition, the distribution of test scores is balanced across groups (Figure [A.1](#page-55-0) shows the CDFs of test scores are similar across groups). We were able to track 88% of students in the non-incentivised test sample at the end of the second year, with no differential attrition. On the incentivised tests, attendance in Levels and Pay for Percentile schools was higher in the second year (Table [A.1\)](#page-55-1). Thus, we present [Lee](#page-51-7) [\(2009\)](#page-51-7) bounds for the treatment effects on incentivised tests.
Table 2: Summary statistics across treatment groups at baseline (February 2015)
| | (1)
Control | (2)
P4Pctile | (3)
Levels | (4)
p-value
(all equal) | |
|-------------------------------|----------------|-----------------|---------------|-------------------------------|--|
| Panel A: Students | | | | | |
| Poverty index (PCA) | 0.01 | -0.08 | 0.01 | 0.42 | |
| | (1.99) | (1.94) | (1.98) | | |
| Age | 8.88 | 8.94 | 8.89 | 0.35 | |
| | (1.60) | (1.67) | (1.60) | | |
| Male | 0.50 | 0.48 | 0.51 | 0.05∗ | |
| | (0.50) | (0.50) | (0.50) | | |
| Kiswahili test score | -0.00 | 0.01 | 0.01 | 0.14 | |
| | (1.00) | (0.99) | (0.98) | | |
| English test score | 0.00 | 0.04 | -0.02 | 0.71 | |
| | (1.00) | (1.03) | (1.04) | | |
| Math test score | -0.00 | -0.01 | -0.01 | 0.56 | |
| | (1.00) | (1.04) | (1.00) | | |
| Tested in yr0 | 0.91 | 0.89 | 0.90 | 0.41 | |
| | (0.29) | (0.31) | (0.30) | | |
| Tested in yr1 | 0.87 | 0.87 | 0.88 | 0.20 | |
| | (0.33) | (0.34) | (0.32) | | |
| Tested in yr2 | 0.88 | 0.88 | 0.89 | 0.56 | |
| | (0.33) | (0.32) | (0.32) | | |
| Panel B: Schools | | | | | |
| Total enrollment | 643.42 | 656.35 | 738.37 | 0.67 | |
| | (331.22) | (437.74) | (553.33) | | |
| Facilities index (PCA) | 0.18 | -0.11 | -0.24 | 0.07∗ | |
| | (1.23) | (0.97) | (1.01) | | |
| Urban | 0.15 | 0.13 | 0.17 | 0.92 | |
| | (0.36) | (0.34) | (0.38) | | |
| Single shift | 0.63 | 0.62 | 0.62 | 0.95 | |
| | (0.49) | (0.49) | (0.49) | | |
| Panel C: Teachers (Grade 1-3) | | | | | |
| Male | 0.42 | 0.38 | 0.35 | 0.19 | |
| | (0.49) | (0.49) | (0.48) | | |
| Age (Yrs) | 37.89 | 37.02 | 37.70 | 0.18 | |
| | (11.35) | (11.23) | (11.02) | | |
| Tertiary education | 0.87 | 0.88 | 0.87 | 0.74 | |
| | (0.33) | (0.32) | (0.33) | | |
| | | | | | |
This table presents the mean and standard error of the mean (in parentheses) for several characteristics of students (Panel A), schools (Panel B), and teachers (Panel C) across treatment groups. Column 4 shows the p-value from testing whether the mean is equal across all treatment groups (*H*0 := mean is equal across groups). The p-value is for a test of equality of means, after controlling for the stratification variables used during randomisation. The poverty index is the first component of a Principal Component Analysis (PCA) of the following assets: mobile phone, watch/clock, refrigerator, motorbike, car, bicycle, television, and radio. The school facilities index is the first component of a Principal Component Analysis (PCA) of indicator variables for outer wall, staff room, playground, library, and kitchen. Standard errors are clustered at the school level for the test of equality. ∗ *p* < 0.10, ∗∗ *p* < 0.05, ∗∗∗ *p* < 0.01
# **3.3 Empirical Specification**
We estimate the effect of our interventions on students' test scores using the following OLS equation:
$$Z_{isdt} = \delta_0 + \delta_1 Levels_s + \delta_2 P4Pctile_s + \delta_3 Z_{isd,t=0} + X_i \delta_4 + X_s \delta_5 + \gamma_d + \gamma_g + \varepsilon_{isdt}, \tag{2}$$
where *Zisdt* is the test score of student *i* in school *s* in district *d* at the end of year *t*. *Levels* and *P*4*Pctile* are binary variables which capture the treatment assignment of each school. *Xi* is a series of student characteristics (age, gender, and grade), *Xs* is a set of school characteristics including facilities, students per teacher, school committee characteristics, average teacher age, average teacher experience, average teacher qualifications, the fraction of female teachers, and the stratification dummies. *γd* is a set of district fixed effects, and *γg* is a set of grade fixed effects.
We scale our test scores using an Item Response Theory (IRT) model and then normalise them using the mean and standard deviation of the control schools to facilitate a clear interpretation of our results. We include baseline test scores and district fixed effects in our specifications to increase precision.[30](#page-21-0)
We examine the incentives' impact using both the non-incentivised and incentivised testing data. However, given the limited student characteristics in the incentivised test data, this analysis includes fewer student-level controls. We use a similar specification to examine teachers' behavioural responses.
# **4 Results**
In this section, we first explore how both incentive systems affected student test scores and grade repetition. We then examine whether the incentives increase observable teacher effort or change teacher behaviour. We then turn to heterogeneity by student and teacher characteristics. Finally, we explore possible mechanisms that could explain our results on test scores. Replication files are available at [Mbiti](#page-52-0) *[et al.](#page-52-0)* [\(2022\)](#page-52-0).
30We also balanced the timing of our survey activities, including the non-incentivised tests, across treatment arms. Hence, the results are not driven by imbalanced survey timing.
# **4.1 Test Scores**
We present the estimated treatment effects of the incentive programs on student learning using data from both the non-incentivised test (Table [3,](#page-25-0) Panel A) and the incentivised test (Table [3,](#page-25-0) Panel B). As discussed earlier, we focus our main analysis on math and Kiswahili due to the curriculum changes. We provide estimates of the intervention on English test scores in Table [A.2](#page-56-0) in Appendix [A.](#page-55-2) To address multiple testing concerns, we also present estimates for a composite index of learning computed using an Item Response Theory model (Columns 3 and 6 in Table [3\)](#page-25-0).
In the first year, both incentive schemes resulted in small but imprecisely estimated changes in test scores on the non-incentivised test. Focusing on the composite learning index (Panel A, Column 3), test scores increased by about 0.057*σ* (p-value 0.23) in Levels schools relative to the control group. Pay for Percentile schools scored 0-.027*σ* (p-value 0.48) below control schools. In the second year of the program, the estimated treatment effects on the non-incentivised test are generally larger than the first-year estimates (Panel A, Columns 4-6). Test scores on the composite index increased by 0.095*σ* (p-value 0.036) in Levels schools and 0.041*σ* (p-value 0.33) in Pay for Percentile schools.[31](#page-22-0)
Most of the existing literature on teacher incentives relies on data from incentivised tests that are used to determine teacher rewards [\(Muralidharan and Sundararaman,](#page-52-3) [2011;](#page-52-3) [Fryer,](#page-49-3) [2013;](#page-49-3) [Neal and Schanzenbach,](#page-52-7) [2010\)](#page-52-7). Following this practice, we also present the treatment effects of our interventions using incentivised exams (Panel B). Generally, the estimated treatment effects are larger compared to those estimated using the non-incentivised test (Panel A). In the first year of the program our composite measure of learning was 0.17*σ* higher (p-value 0< 0.01) in Levels schools relative to the control group and 0.059*σ* higher in Pay for Percentile schools, but this was not statistically significant (p-value 0.28, see Column 3). In the second year, learning was 0.22*σ* higher (p-value < 0.01) in Levels schools and 0.13*σ* higher (p-value 0.027) in Pay for Percentile schools.[32](#page-22-1)
31Unlike other settings (e.g., [Macartney](#page-51-8) [\(2016\)](#page-51-8)), there are no dynamic incentives in either treatment. In the Levels scheme, payments do not depend on previous student performance. In the Pay for Percentile scheme, while the initial seeding for each student depends on past performance, teachers typically teach a single grade. Thus, since they get a new set of students each year, behaviour today does not impact payments in the future.
32The treatment effects on threshold specific pass rates are shown in Tables [A.3-](#page-57-0) [A.6](#page-59-0) in Appendix [A.](#page-55-2)
The estimated treatment effects (on the incentivised test) for Levels schools are comparable with those found in previous experiments in India and Mexico [\(Muralidharan and Sundararaman,](#page-52-3) [2011;](#page-52-3) [Behrman](#page-47-6) *[et al.](#page-47-6)*, [2015\)](#page-47-6). The estimated effects for the Pay for Percentile design are lower than those found in [Loyalka](#page-51-2) *[et al.](#page-51-2)* [\(2019\)](#page-51-2) but larger than those in [Gilligan](#page-49-7) *et al.* [\(2019\)](#page-49-7).[33](#page-23-0) The results suggest that the Levels design performs at least as well as the Pay for Percentile design. Despite the theoretical predictions, we find no evidence that the Pay for Percentile system leads to larger increases in learning relative to Levels. Focusing on the composite test scores, the estimated differences between the incentive designs (*α*3 and *β*3 in Columns 3 and 6) are always negative (i.e., Levels mostly outperforms Pay for Percentile) and statistically significant in three out of four cases.
The larger treatment effects found in the incentivised test are likely driven by test-taking effort, where teachers had incentives to motivate their students to take the tests seriously. The importance of student test-taking effort has been documented in other settings, such as an evaluation of teacher and student incentives in Mexico City [\(Behrman](#page-47-6) *et al.*, [2015\)](#page-47-6). As described in Section [3.2](#page-18-0) and Appendix [E,](#page-106-0) our implementation team tightly controlled the administration of the incentivised test, mitigating concerns about cheating. Assuming that test-taking effort drives all the differences between our incentivised and non-incentivised results, student effort can increase test scores between 0.0051*σ* and 0.11*σ* (see Panel C). This is generally in line with the findings of [Levitt](#page-51-5) *et al.* [\(2016\)](#page-51-5) and [Gneezy](#page-49-5) *et al.* [\(2019\)](#page-49-5).[34](#page-23-1)
Given the reward structure, teachers in both treatment arms were motivated to ensure that their students took the incentivised test. There were no incentives to exclude academically weaker students because learning gains from all students would be rewarded. In the second year of the study, teachers in the Levels schools increased student participation in the incentivised test by 5 percentage points. Their counterparts in Pay for Percentile schools increased participation by 3 percentage points (see Table [A.1](#page-55-1) in Appendix [A\)](#page-55-2). Following [Lee](#page-51-7) [\(2009\)](#page-51-7), we compute bounds on the treatment effects by trimming the excess test-takers from the left and right tails of the incentivised test distribution. Focusing on the year-two results for
33For the full sample [Gilligan](#page-49-7) *et al.* [\(2019\)](#page-49-7) find that Pay for Percentile incentives have a small (0.01*σ*) and statistically insignificant effect on learning. However, there is substantial heterogeneity in treatment effects. Pay for Percentile incentives improve learning outcomes in schools with books by 0.11*σ* on the grade-relevant sub-test. In schools without books, there is no significant treatment effect on learning.
34The difference between the treatment effects in the incentivised and non-incentivised test across treatments arms is not statistically significant (i.e., testing *γ*1 = *γ*2 and *γ*1 = *γ*2 = *γ*3). We can match some students across both tests. Although matching is not random (see Table [A.8\)](#page-61-0), the results are qualitatively similar if we focus on this sample (see Table [A.9\)](#page-62-0).
brevity, the 95% confidence interval for the treatment effects from this bounding exercise for math is from -0.023 to 0.32 in the Levels treatment and 0.014 to 0.17 in the Pay for Percentile treatment. The bounds for Kiswahili range from 0.027 to 0.35 in the Levels and -0.0032 to 0.17 in the Pay for Percentile (see Table [A.7](#page-60-0) in Appendix [A\)](#page-55-2).
As discussed previously, we had limited information to properly group grade 1 students in Pay for Percentile schools. As this may limit the effectiveness of the Pay for Percentile scheme, we examine the effects of our interventions by focusing on grade 2 and 3 students, where we can appropriately group most students by ability. Our results are generally robust to this sample restriction (see Table [A.10](#page-63-0) in Appendix [A\)](#page-55-2).
Finally, the results are robust to different controls. Our results are qualitatively similar whether we use a parsimonious specification that only includes randomisation strata (see Table [A.11\)](#page-64-0) or specifications that use controls selected by a post-double lasso procedure (see Tables [A.12](#page-65-0)[-A.13\)](#page-66-0).
Table 3: Effect on test scores
| | (1) | (2)
Year 1 | (3) | (4) | (5)
Year 2 | (6) |
|---------------------------------------------|----------|---------------|----------|--------|---------------|----------|
| | Math | Kiswahili | Combined | Math | Kiswahili | Combined |
| Panel A: Non-incentivi | sed test | | | | | |
| Levels $(\alpha_1)$ | .039 | .045 | .057 | .068* | .096* | .095** |
| , -/ | (.047) | (.048) | (.047) | (.04) | (.052) | (.045) |
| P4Pctile ( $\alpha_2$ ) | 015 | 033 | 027 | .072** | .0018 | .041 |
| \ - / | (.04) | (.039) | (.039) | (.037) | (.05) | (.043) |
| N. of obs. | 4,781 | 4,781 | 4,781 | 4,869 | 4,869 | 4,869 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 053 | 078* | 084* | .0047 | 094* | 053 |
| p-value ( $H_0$ : $\alpha_3 = 0$ ) | .23 | .084 | .056 | .92 | .074 | .27 |
| Panel B: Incentivised to | est | | | | | |
| Levels $(\beta_1)$ | .11** | .13*** | .17*** | .14*** | .18*** | .22*** |
| ų -, | (.047) | (.048) | (.064) | (.045) | (.046) | (.059) |
| P4Pctile ( $\beta_2$ ) | .066* | .017 | .059 | .093** | .085* | .13** |
| • | (.039) | (.043) | (.054) | (.04) | (.045) | (.056) |
| N. of obs. | 48,077 | 48,077 | 48,077 | 59,680 | 59,680 | 59,680 |
| $\beta_3 = \beta_2 - \beta_1$ | 047 | 11** | 11* | 044 | 093** | 096* |
| p-value ( $H_0: \beta_3 = 0$ ) | 0.30 | 0.026 | 0.070 | 0.31 | 0.045 | 0.097 |
| Panel C: Incentivised - | Non-inc | entivised | | | | |
| $\gamma_1 = \beta_1 - \alpha_1$ | .065 | .075 | .1 | .063 | .072 | .11 |
| $p$ -value( $\gamma_1 = 0$ ) | .13 | .1 | .046 | .12 | .12 | .025 |
| $\gamma_2 = \beta_2 - \alpha_2$ | .075 | .044 | .078 | .019 | .077 | .077 |
| $p$ -value( $\gamma_2 = 0$ ) | .083 | .32 | .14 | .64 | .073 | .11 |
| $\gamma_3 = \beta_3 - \alpha_3$ | .01 | 031 | 024 | 044 | .0058 | 037 |
| $p$ -value( $\gamma_3 = 0$ ) | .81 | .51 | .65 | .28 | .91 | .49 |
| $\gamma_1 - \gamma_2$ | 01 | .031 | .024 | .044 | 0058 | .037 |
| p-value( $\gamma_1 - \gamma_2 = 0$ ) | .81 | .51 | .65 | .28 | .91 | .49 |
| $\gamma_1 - \gamma_3$ | .075 | .044 | .078 | .019 | .077 | .077 |
| p-value( $\gamma_1 - \gamma_3 = 0$ ) | .083 | .32 | .14 | .64 | .073 | .11 |
| $\gamma_2 - \gamma_3$ | .085 | .013 | .055 | 026 | .083 | .041 |
| p-value( $\gamma_2 - \gamma_3 = 0$ ) | .26 | .86 | .56 | .71 | .31 | .65 |
| p-value( $\gamma_1 = \gamma_2 = \gamma_3$ ) | .17 | .25 | .12 | .28 | .13 | .062 |
Results from estimating Equation 2 for different subjects at both follow-ups. Panel A uses data from the non-incentivised test taken by a sample of students. Panel B uses data from the incentivised test taken by all students. Control variables in both panels include student characteristics (gender and grade) and school characteristics (PTR, Infrastructure Principal Component Analysis (PCA) index, a PCA index of how close the school is to different facilities, and an indicator for whether the school is single shift or not). Panel C tests the difference between the treatment estimates in panels A and B. Table A.11 provides a version without school controls. Tables A.12-A.13 provide results when post double lasso selection is used to select the control variables. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\*\* p < 0.05, \*\*\*\* p < 0.01
# 4.2 Grade repetition
Cross-country comparisons reveal a negative correlation between income per capita and the grade repetition rate in primary school (Manacorda, 2012). Grade repetition is commonplace in developing countries
and is thought to impose significant individual and social costs, such as an increase in the probability that a student drops out of school [\(Manacorda,](#page-51-9) [2012\)](#page-51-9). Thus, lowering repetition rates can improve the fiscal efficiency of schools.
In Tanzania, the introduction of the 3R (Reading, wRiting, and aRithmetic) curriculum in 2015 was accompanied by a change in grade repetition policy in grades 1, 2, and 3. As a result, promotion is no longer automatic, and pupils can be forced to repeat based on a decision by the school committee (automatic promotion remains in place after grade 3). School committees use internal data to make decisions (not data collected from the study), but these data and decision-making processes are not standardised.
We examine the impact of both treatments on grade repetition in Table [4.](#page-27-0) In 2015, the first year of both the incentive program and the new retention policy, we do not find any statistically significant changes in repetition rates in Levels or Pay for Percentile schools (Column 1). At the end of the second year, repetition rates in Levels schools were 3.3 percentage points lower than the control group (p-value 0.048), a 24% reduction. There was a small positive and statistically insignificant effect on grade repetition among students in Pay for Percentile schools. Formal hypothesis tests show that the estimated reduction in Levels schools was significantly lower (p-value 0.034) compared to the estimated change in Pay for Percentile schools.
Table 4: Effect on grade repetition
| | (1) | (2) |
|----------------------------------|--------|--------|
| | Year 1 | Year 2 |
| Levels $(\alpha_1)$ | 0095 | 033** |
| | (.02) | (.017) |
| P4Pctile ( $\alpha_2$ ) | .025 | .0025 |
| | (.017) | (.014) |
| N. of obs. | 4,781 | 4,869 |
| Mean control | .13 | .14 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | .035* | .035** |
| p-value ( $H_0: \alpha_3 = 0$ ) | .062 | .034 |
Results from estimating Equation 2 for whether a student is in a lower grade than expected at the end of the first year (Column 1) and at the end of the second year (Column 2). Control variables include student characteristics (age, gender, grade, and lag test scores) and school characteristics (PTR, Infrastructure PCA index, a PCA index of how close the school is to different facilities, and an indicator for whether the school is single shift or not). Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
# 4.3 Spillovers to Other Grades and Subjects
As the teacher incentives only covered numeracy and literacy in grades 1, 2, and 3, a potential concern is that teachers and schools focus on these grades and subjects to the detriment of other grades and subjects. For example, schools may shift resources such as textbook purchases from higher grades to grades 1, 2, and 3. In addition, teachers may cut back on teaching non-incentivised subjects such as science. On the other hand, if our incentive programs improve literacy and numeracy skills, they may promote student learning in other subjects, and these gains may persist over time. To assess possible spillovers, we examine test scores in science for grades 1, 2, and 3. We also examine test scores in grade 4 to test for any negative spillovers in higher grades and the persistence of any learning gains induced by the program (in the second year of the evaluation).
Overall, we do not see decreases in fourth graders' test scores, which suggests that schools were not disproportionately shifting resources away from higher grades (Table 5, Panel A). In the first year of the program, composite test scores for grade 4 students in Levels schools increased by $0.1\sigma$ (p-value 0.045) (Column 3). In Pay for Percentile schools, we find relatively small (-.024 $\sigma$ ) and statistically insignificant
(p-value 0.63) effects on composite test scores. Since we tested fourth-grade students and collected information on them at baseline, we conjecture that fourth-grade teachers assumed they would be included in the incentives. As a result of this belief, they may have exerted effort in the first year but not in the second year once their non-eligibility had been confirmed. This type of spillover was also documented by [Kremer](#page-50-7) *et al.* [\(2009\)](#page-50-7), where a student incentive program for girls improved the performance of non-eligible boys who believed they would also benefit from the program.[35](#page-28-0)
As third graders in the first year of our program transitioned to the fourth grade in the second year of the program, the fourth-grade results in the second year suggest that the learning gains from both incentive programs fade over time (Table [5,](#page-29-0) Panel A, Columns 4 to 6).
Contrary to the concerns of teacher performance pay critics, the effects of both programs on science test scores are generally positive, suggesting that any estimated gains attributable to the incentives are not coming at the expense of learning in other subjects or domains that are not directly incentivised (see Table [5,](#page-29-0) Panel B).
35We also test for any spillover effects on the seventh-grade primary school national exit exam (PSLE). We do not find evidence that our incentives affected students' performance on those tests. See Table [A.14](#page-67-0) in Appendix [A.](#page-55-2)
Table 5: Spillovers to other grades and subjects
| Panel A: Grade 4 | | | | | | |
|----------------------------------|--------|---------------|----------|--------|---------------|----------|
| | (1) | (2)
Year 1 | (3) | (4) | (5)
Year 2 | (6) |
| | Math | Kiswahili | Combined | Math | Kiswahili | Combined |
| Levels $(\alpha_1)$ | .13** | .044 | .1** | .059 | .042 | .049 |
| | (.062) | (.051) | (.05) | (.062) | (.057) | (.053) |
| P4Pctile ( $\alpha_2$ ) | 03 | 033 | 024 | 0038 | .00015 | 0028 |
| | (.054) | (.054) | (.05) | (.06) | (.051) | (.05) |
| N. of obs. | 1,513 | 1,513 | 1,513 | 1,482 | 1,482 | 1,482 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 16** | 077 | 13** | 063 | 041 | 052 |
| p-value ( $H_0: \alpha_3 = 0$ ) | .011 | .13 | .017 | .27 | .44 | .32 |
Panel B: Science (Grades 1-3)
| | Year 1 | Year 2 |
|----------------------------------|--------|--------|
| Levels $(\alpha_1)$ | .069 | .083 |
| | (.063) | (.06) |
| P4Pctile ( $\alpha_2$ ) | 002 | .078 |
| | (.05) | (.057) |
| N. of obs. | 4,781 | 4,869 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 071 | 0055 |
| p-value ( $H_0: \alpha_3 = 0$ ) | .26 | .92 |
Results from estimating Equation 2 for grade 4 students (Panel A) and for grade 1-3 students in science (Panel B). Control variables include student characteristics (age, gender, grade, and lag test scores) and school characteristics (PTR, Infrastructure PCA index, a PCA index of how close the school is to different facilities, and an indicator for whether the school is single shift or not). Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
# 4.4 Equity concerns
In this section, we explore the heterogeneity in treatment effects across the distribution of student baseline composite test scores in Figure 2 (for the non-incentivised tests) and Figure 3 (for the incentivised tests).36 This allows us to examine where teachers differentially focus their efforts in the distribution of student baseline test scores. The analysis also allows us to gauge whether the Levels system exacerbates inequality relative to the Pay for Percentile System, as suggested by Macartney *et al.* (2021).
Despite equity concerns, we do not find evidence that the Levels system increases inequality relative to the Pay for Percentile System. In both years, we find similar patterns of heterogeneity in treatment effects by student baseline test scores using data from the incentivised and non-incentivised tests. In the first year
<sup>36We also explore heterogeneity by additional student characteristics such as gender, as well as school characteristics such as pupil-teacher ratio, and find limited evidence of heterogeneity in those characteristics (see Tables A.15 and A.16 in Appendix A for details).
of the program, teachers in both systems focused their attention on the best students. This pattern is more pronounced in Pay for Percentile schools, where we can reject that the estimated learning gains are the same for all quintiles (p-value 0.016). In the second year of the program, the treatment effects were more balanced across the distribution of students, and we fail to reject the hypothesis that the treatment effects for each baseline score quintile are equal.[37](#page-30-0) Our first-year results for the Pay for Percentile treatment are in line with [Gilligan](#page-49-7) *et al.* [\(2019\)](#page-49-7), who find that learning gains were greater for above-median students, especially in schools with books. Our second-year results for the Pay for Percentile treatment align with [Loyalka](#page-51-2) *et al.* [\(2019\)](#page-51-2) who find learning gains across the entire distribution of students.
We also examine if teachers in Levels schools targeted students near passing thresholds. If this were the case, we would expect bunching around the thresholds set by the Levels system. However, the general pattern we observe is that most students are either well above or well below the passing threshold, with no bunching at the threshold (see Figures [A.6](#page-74-0) and [A.7\)](#page-75-0). Combined with our heterogeneity analysis by students' baseline ability, these results allow us to rule out this possibility.
Finally, both treatments reduce the standard deviation and the Gini coefficient of test scores within a classroom (see Appendix [A.13\)](#page-76-0). In some cases, the reduction in inequality of test scores is greater for the Levels system, although the difference is not statistically significant nor consistent across years, subjects, or measures of inequality.
37Subject specific results are available in Figures [A.2](#page-69-0) - [A.5](#page-72-0) in Appendix [A.](#page-55-2)
(a) Year 1
(b) Year 2
Note: These figures show the treatment effects (and their 95% confidence interval) in the composite score of the non-incentivised test (y-axis) at the end of the first year (2a) and the second year (2b) by student's quintile in the baseline score (x-axis). We also show p-values for testing whether the treatment effects for students in the first quintile are the same as the effects for those in the last quintile, and for testing whether the treatment effect is the same across all five quintiles.
## (a) Year 1
(b) Year 2
Note: These figures show the treatment effects (and their 95% confidence interval) in the composite score of the incentivised test (y-axis) at the end of the first year (2a) and the second year (2b) by student's quintile in the baseline score (x-axis). We also show p-values for testing whether the treatment effects for students in the first quintile are the same as the effects for those in the last quintile, and for testing whether the treatment effect is the same across all five quintiles.
# **4.5 Heterogeneity by Teacher Characteristics**
Empirical evidence shows that women are more averse to competition and exert relatively less effort than men in competitive situations such as rank-order tournaments [\(Niederle and Vesterlund,](#page-53-9) [2007,](#page-53-9) [2011\)](#page-53-10). However, we do not find any significant heterogeneous treatment effects by teacher gender (Table [6,](#page-34-0) Column 1). We also do not find any heterogeneous effects by teacher's age, which proxies for experience.
Although previous studies (e.g., [Metzler and Woessmann](#page-52-9) [\(2012\)](#page-52-9) and [Bietenbeck](#page-47-7) *et al.* [\(2018\)](#page-47-7)) have shown that teacher content knowledge is predictive of student learning outcomes, we do not find any significant heterogeneity in our treatment effects by teacher content knowledge, which our survey team measured through math and word association tests (Column 3). More effective teachers, as measured by the headteacher's performance rating, as well as teachers who were more confident in their teaching abilities, responded more to both incentives (Columns 4 and 5).
| | (1) | (2) | (3) | (4) | (5) |
|--------------------|---------|-----------|-------------|-----------|-------------|
| | Male | Age | IRT | HT Rating | Self Rating |
| Levels | 0.056 | 0.070 | $0.066^{*}$ | 0.091** | 0.072** |
| | (0.039) | (0.052) | (0.037) | (0.038) | (0.036) |
| Gains | -0.0066 | -0.000058 | 0.010 | 0.035 | 0.014 |
| | (0.033) | (0.047) | (0.033) | (0.039) | (0.030) |
| Levels*Covariate | 0.0031 | -0.00025 | 0.016 | 0.099*** | 0.062** |
| | (0.056) | (0.0012) | (0.033) | (0.022) | (0.030) |
| P4Pctile*Covariate | 0.035 | 0.00025 | 0.0081 | 0.048* | 0.091*** |
| | (0.052) | (0.0012) | (0.036) | (0.027) | (0.029) |
| Covariate | -0.066 | -0.00022 | -0.017 | -0.037** | -0.077*** |
| | (0.041) | (0.0013) | (0.025) | (0.018) | (0.021) |
| Covariate mean | .37 | 39 | 15 | 012 | 074 |
| N. of obs. | 19,300 | 19,300 | 19,300 | 9,738 | 19,300 |
Table 6: Heterogeneity by teacher characteristics
The outcome variables are student test scores. The data is at the student-subject-year level and pools both follow-ups and both subjects (Kiswahili and math). Each column shows the heterogeneous treatment effect by different teacher characteristics: sex (Column 1), age (Column 2), content knowledge scaled by an IRT model (Column 3), headteacher rating (Column 4) only requested for math and Kiswahili teachers at the end of the second year — and self-rating (Column 5), collected at the end of the school year in both years. In addition to the variables shown, all our specifications include both treatment indicators and the covariate. We use three measures of teacher ability to explore the heterogeneity in treatment effects. First, teachers were tested on all three subjects, and we created an index of content knowledge using an IRT model. Second, headteachers were asked to rate teacher performance in seven dimensions, including the ability to ensure that students learn and classroom management skills. Third, to create the self-perception metric, we create a Principal Component Analysis (PCA) index based on teacher responses to the following five statements: "I am capable of motivating students who show low interest in school," "I am capable of implementing alternative strategies in my classroom," "I am capable of getting students to believe they can do well in school," "I am capable of assisting families in helping their children do well in school," and "I am capable of providing an alternative explanation for example when students are confused." Standard errors, clustered at the school level, are in parentheses. p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
## 4.6 Teacher Effort and Behaviour
Since the treatments were designed to elicit teacher effort and modify their behaviour, we examine teacher responsiveness to the incentives in this section. We measure teacher effort using four different approaches and report the second-year estimates for brevity (Appendix A reports the first-year estimates). First, our survey team measured and collected teacher presence data shortly after our team arrived at a school in the morning. Overall, we do not find any effect in this dimension of teacher effort across our treatments (see Table 7, Panel A).38
<sup>38Our results suggest a (small) reduction in school absenteeism, and an increase in classroom presence, but these coefficients are imprecisely estimated. Getting precision on this particular measure would require coordinating multiple surprise visits, which is often impractical due to budget constraints and other logistical challenges (Muralidharan, 2017). Our budget did not allow us to
Second, we use data from student reports to examine additional dimensions of teacher effort (see Panel B in Table 7). We find suggestive evidence that teachers in treatment schools are less likely to hit students and more likely to call them by their names during class time and provide additional help outside the classroom. However, teachers did not increase the amount of homework assigned to students.
Table 7: Treatment effects on teacher behaviour - Year 2
| Panel A: Spot checks | | |
|----------------------------------|------------------|---------------------|
| | (1)
In school | (2)
In classroom |
| Levels $(\alpha_1)$ | -0.025 | 0.025 |
| | (0.050) | (0.053) |
| P4Pctile ( $\alpha_2$ ) | -0.0050 | 0.023 |
| | (0.044) | (0.044) |
| N. of obs. | 180 | 180 |
| Mean control | .7 | .36 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | .02 | 0021 |
| p-value ( $H_0: \alpha_3 = 0$ ) | .71 | .97 |
Panel B: Student reports
| | (1)
Extra help | (2)
Homework | (3)
Call by name | (4)
Hit |
|----------------------------------|-------------------|-----------------|---------------------|------------|
| Levels $(\alpha_1)$ | 0.0052 | 0.0029 | 0.080** | -0.030 |
| | (0.0096) | (0.018) | (0.037) | (0.035) |
| P4Pctile ( $\alpha_2$ ) | $0.016^{*}$ | -0.023 | 0.047 | -0.061* |
| | (0.0097) | (0.019) | (0.032) | (0.032) |
| N. of obs. | 9,557 | 9,557 | 9 , 557 | 9,557 |
| Mean control | .062 | .12 | .5 | .37 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | .011 | 026 | 032 | 031 |
| p-value ( $H_0: \alpha_3 = 0$ ) | .29 | .24 | .34 | .35 |
Panel A presents school-level data on teacher absenteeism (Column 1) and time-on-task (Column 2). Panel B presents student-level data on teacher behaviour (as reported by students), extra help (Column 1), homework assignment (Column 2), calling by name (Column 3), and hitting/pinching/slapping students (Column 4). This table uses data collected at the end of the second school year of the experiment. Table A.19 presents results using data collected at the end of the first school year of the experiment. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
As our third measure of teacher effort, we use two sets of classroom observations. We conducted "external classroom observations," where our survey teams observed teacher behaviour by standing outside the classroom for several minutes to prevent disruptions.39 We also conducted within-classroom observations
collect data in this manner. Instead, these data were collected during announced visits because our main goal was to test students and survey teachers with minimal attrition.
<sup>39Schools in Tanzania have open layouts where classrooms are built around an open space in the middle. This layout allows surveyors to stand in the open space and observe the class from a distance through the windows.
following the World Bank Service Delivery Indicator protocols. However, in-class observations are often affected by Hawthorne/John Henry effects, reducing how useful these protocols are (Muralidharan and Sundararaman, 2010). Even though the external observations are less detailed, they are arguably better able to capture broad measures of teacher behaviour because they are not affected by Hawthorne/John Henry effects.
Our findings using the external observations are shown in Table 8. Students in both Levels and Pay for Percentile schools were less likely to be off-task during the classroom observations. However, the reduction is only statistically significant in Pay for Percentile schools. While there is an increase in the likelihood that teachers are teaching (instead of off-task or managing the classroom), this is imprecisely estimated and statistically insignificant for both treatments. The results from the in-class observations (see Appendix Table A.21) also suggest teachers are less likely to be off-task and more likely to be teaching. However, these results are also imprecisely estimated and statistically insignificant for both treatments, perhaps a reflection of Hawthorne/John Henry effects.
Table 8: External classroom observation - Year 2
| | (1)
Teaching | (2)
Classroom management | (3)
Teacher off task | (4)
Student off task |
|----------------------------------|------------------|-----------------------------|-------------------------|-------------------------|
| Levels $(\alpha_1)$ | 0.058
(0.054) | -0.028
(0.022) | -0.032
(0.056) | -0.036
(0.042) |
| P4Pctile ( $\alpha_2$ ) | 0.024
(0.050) | -0.063***
(0.023) | 0.035
(0.052) | -0.073**
(0.033) |
| N. of obs. | 772 | 772 | 772 | 772 |
| Control mean | .69 | .041 | .27 | .048 |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 033 | 035* | .067 | 038 |
| p-value ( $H_0: \alpha_3 = 0$ ) | .55 | .095 | .22 | .28 |
The outcome variables in this table come from independent classroom observations performed by the research team for several minutes before teachers noticed they were being observed. Teachers are classified as doing one of three activities: Teaching (Column 1), managing the classroom (Column 2), and being off-task (Column 3). If students are distracted, we classify the class as having students off-task (Column 4). This table uses data collected at the end of the second school year of the experiment. Table A.20 presents results using data collected at the end of the first school year of the experiment. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
As our final measure of teacher effort, we use data from inspections of student notebooks.[40](#page-37-0) During these inspections, our enumerators checked whether students had recent assignments and whether they had received feedback from the teacher. These data corroborate the student reports that teachers did not assign more homework in response to the incentives. However, there is some suggestive evidence from student notebooks that homework assignments were longer in Levels schools (see Table [A.22](#page-80-0) Column 3), but this might reflect multiple testing within this family of outcomes. These data also highlight ample room for improvement among teachers in their grading and feedback practices. Only two-thirds of assignments were graded, and one-fifth of notebooks had written comments or feedback.
In short, despite comprehensive data on teacher absenteeism, time-on-task (measured through classroom observations), student reports on teacher behaviour, and data from student notebooks, we are only able to find suggestive evidence of changes in teacher effort in a narrow set of outcomes. This could be explained by the fact that teachers can adjust effort on many margins (e.g., pedagogy, time-on-task, homework, socio-emotional support) in response to incentives linked to student learning, and these margins are either difficult to measure or unobservable. However, overall, our results suggest minor improvements in teacher effort that are generally similar across both treatment groups.
# **4.7 Earnings Expectations and Turnover**
Before the payout of the bonuses, we collected data on teachers' earnings expectations from the incentives and their beliefs about their performance relative to other teachers in the district. As these questions were only applicable to teachers in the incentive programs, we compare teachers in the Pay for Percentile arm to the Levels scheme, which serves as the omitted category in Table [9.](#page-38-0)
Both Pay for Percentile and Levels teachers overestimated their expected earnings. For example, pay for Percentile teachers expected to earn about 430,000 TZS in bonus payments, while Levels teachers expected about 525,000 TZS. However, the average bonus payment was 226,337 TZS in 2016. Thus, on average, teachers in Levels and Pay for Percentile schools expected to earn 2.3 and 1.9 times the actual average
40We are not aware of other studies exploiting this potential data source. We believe there is great potential in using student notebooks as a data source to measure student and teacher effort.
payment, respectively. If teacher effort mirrors their expectations, this could provide insights into the factors that drove their behavioural responses.
Teachers in Pay for Percentile schools had lower bonus earnings expectations than their peers in the Levels system. They expected almost 95,000 TZS (US\$42) less in bonus payments than teachers in the Levels system, an 18% reduction in bonus expectations relative to the mean expectations of teachers in the Levels system (Column 1).[41](#page-38-1) The lower expectations among Pay for Percentile teachers could be driven by the greater uncertainty of earnings in rank-order tournaments such as Pay for Percentile systems.[42](#page-38-2) While competitive pressure can be motivating, it can also be demotivating if an individual teacher has low subjective beliefs about their probability of winning relative to the probability of competitors winning.
We also examine differences in teachers' beliefs about their relative ranking within their district based on their (expected) bonus winnings in columns 2 to 4. Overall, we do not find any differences in teachers' beliefs about their rankings across the treatments. Teachers were optimistic about their projected earnings: Less than 1% of teachers expected to be among the bottom earners (Column 2) and 7% were worried about earning a low bonus (Column 5). On the other hand, 80% expected to be among the district's top earners (Column 4).[43](#page-38-3)
Table 9: Teachers' earning expectations
| | Bonus (TZS) | Bottom of the
district | Middle of the
district | Top of the
district | Worried low bonus |
|---------------|-------------|---------------------------|---------------------------|------------------------|-------------------|
| | (1) | (2) | (3) | (4) | (5) |
| P4Pctile (α2) | -94,330∗∗ | .0058 | 012 | .035 | 02 |
| | (37,169) | (.008) | (.04) | (.045) | (.026) |
| N. of obs. | 653 | 676 | 676 | 676 | 676 |
| Mean Levels | 525,641 | .0057 | .11 | .8 | .074 |
This table shows the effect of treatment on teacher self-reported expectations: the expected payoff (Column 1), the expected relative ranking in the district (Columns 2-4), and whether the teacher is worried about receiving a low bonus payment (Column 5). Standard errors, clustered at the school level, are in parentheses. ∗ *p* < 0.10, ∗∗ *p* < 0.05, ∗∗∗ *p* < 0.01
41As in [Brown and Andrabi](#page-48-8) [\(2020\)](#page-48-8), expectations do not track closely to actual performance.
42The shape of the distribution of expected earnings is similar across treatment arms, with a mean-shift but similar variance. See Figure [A.8.](#page-80-1)
43We also explore the extent to which the incentives affected teachers' goal-setting behaviour in Appendix [A.18.](#page-81-0)
One of the potential effects of larger incentives is increasing teacher assignment retention (whether a teacher is still teaching the assigned grade and subject at the end of the year). Teachers can rotate out of their assignments during the year and move to a different grade/subject within the same school or transfer to another school. A teacher who rotates into an incentivised grade/subject combination in a treatment school during the year is eligible for the bonus and is paid a pro-rated bonus. This aspect of the bonus was explained to teachers at baseline.
Teacher assignment retention was slightly higher (around 6 percentage points, from a base of 73%) in the first year in both incentive schools compared to control schools, but the difference is statistically insignificant. In the second year, teacher assignment retention remained 6 percentage points higher in Levels schools (statistically insignificant) compared with control schools. However, assignment retention was 8 percentage points higher (p-value 0.01) in Pay for Percentile schools than control schools by the end of the second year (see Appendix Table [A.24\)](#page-82-0). Higher teacher retention, and thus more experience with the incentive scheme, could explain the larger second-year treatment effect on learning for Pay for Percentile schools (relative to the first year).
# **4.8 Experience with incentive designs**
As discussed in Section [3.2,](#page-18-0) the schools in this study were part of a previous experiment studied by [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3). Although the randomisation of the treatments we study was stratified by the treatment/control assignment in the previous experiment, some of our current results may be driven by exposure to the previous treatment. For instance, teachers who had previously participated in the treatment studied by [Mbiti](#page-51-3) *et al.* [\(2019\)](#page-51-3) might be better able to respond to the current treatments due to their experience with a (single) threshold teacher incentives scheme. We explore this possibility in Tables [A.25-](#page-83-0)[A.26](#page-84-0) in Appendix [A.](#page-55-2) The effects for both treatments are higher for schools that had prior exposure to teacher incentives through the previous experiment. This could reflect learning by doing or greater trust in the system and the implementing team (i.e., teachers are more likely to believe they will get paid for exerting higher effort over time). We cannot reject the null hypothesis that teachers in both systems benefit equally from prior exposure to the incentives.
Due to the churn in teacher grades and school allocations in Tanzania, there is significant variation across schools in the fraction of lower grade teachers who were previously exposed to incentives. We explore this dimension of heterogeneity by interacting the current treatment assignment indicators with a binary variable equal to one if at least 50% of (current) teachers in focal grades and subjects previously participated in an incentive scheme (see Table [10\)](#page-42-0).[44](#page-40-0)
This analysis yields five insights — we focus on the results from the high-stakes test as we have the universe of students and thus more power, but the results are qualitatively similar if we focus on the lowstakes test. First, teachers who were exposed to incentives in the past, but not in the current experiment (i.e., are now in Control schools) do not have a positive treatment effect on test scores, suggesting long-term effects of the previous experiment do not drive our results.[45](#page-40-1)
Second, with the exception of Pay for Percentile teachers who were not previously exposed to incentives in the past, the treatment effects for all other groups are positive, but not all are statistically significant. In the second year, the treatment effects are positive for all groups and all but one are statistically significant on the incentivised test. The pattern for the previously unexposed Pay for Percentile teachers suggests teachers need additional time to react to a scheme without "bright lines", such as Pay for Percentile.
Third, for teachers who had been exposed to incentives in the past, the treatment effects of the Levels and Pay for Percentile treatments are constant in both years of our study (especially in the high-stakes test), suggesting that after two years of exposure to incentives, there is a "steady-state" in the treatment effects.
Fourth, for teachers who had not been exposed to incentives before, treatment effects increase over time and tend to converge to those of previously exposed teachers. This could either reflect learning by doing
44Across schools, the median proportion of teachers who were previously exposed to incentives is 50% in both years of our study. The results are qualitatively similar if we interact the current treatment assignment indicators with a binary variable equal to one if at least 25% of (current) teachers in focal grades and subjects previously participated in an incentive scheme (see Table [A.27\)](#page-85-0).
45Further, test scores in Control schools who were treated in the previous experiment are not different from other Control schools (see Table [A.26\)](#page-84-0). Moreover, if we restrict our analysis to the set of students that were not exposed to the previous treatment (those in Grade 1 in year 1, and those in Grade 1 and 2 in year 2), the results are qualitatively similar (see Table [A.28\)](#page-86-0), suggesting that long-term effects from the treatment in the previous experiment are not driving the results in the experiment we analyse in this article.
(with respect to incentives) or the development of trust between individual teachers and the implementing agency.[46](#page-41-0)
Fifth, the Levels treatment effect is (at least) as effective as the Pay for Percentile effect regardless of whether teachers were exposed to incentives in the past or not.[47](#page-41-1)
46Note the treatment effect grows over time in this experiment, as well as in the previous experiment [\(Mbiti](#page-51-3) *et al.*, [2019\)](#page-51-3), further suggesting it takes some time to build trust and to get familiarised with performance pay schemes.
47Excluding schools that did not receive incentives in the previous experiment from the estimation also yields treatment effects for Levels that are above those of Pay for Percentile schools (see Table [A.29\)](#page-87-0). Since these results focus on schools that were previously exposed to incentives, they are more likely to reflect "steady-state" effects.
Table 10: Heterogeneity by whether at least half of the current teachers were previously exposed to incentives
| | (1) | (2) |
|---------------------------------------------------------------------------|--------|--------|
| | Year 1 | Year 2 |
| Panel A: Non-incentivised test | | |
| Levels $\times$ Teachers not incentivized in previous RCT ( $\alpha_1$ ) | .049 | .11** |
| 1 | (.061) | (.056) |
| Levels $\times$ Teachers incentivized in previous RCT ( $\alpha_2$ ) | .074 | .088 |
| • | (.06) | (.063) |
| P4Pctile × Teachers not incentivized in previous RCT ( $\beta_1$ ) | 063 | .0089 |
| 1 1 1 1 | (.056) | (.05) |
| P4Pctile × Teachers incentivized in previous RCT ( $\beta_2$ ) | .015 | .062 |
| ¥ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | (.049) | (.063) |
| Teachers incentivized in previous RCT | 054 | .024 |
| • | (.063) | (.056) |
| N. of obs. | 4,781 | 4,869 |
| $p$ -value( $H_0: \alpha_1 = \alpha_2$ ) | .75 | .76 |
| $p -value(H_0: \beta_1 = \beta_2)$ | .29 | .49 |
| $p$ -value( $H_0: \alpha_1 = \beta_1$ ) | .059 | .08 |
| $p$ -value( $H_0: \alpha_2 = \beta_2$ ) | .3 | .68 |
| p-value( $H_0: (\alpha_1 - \beta_1) = (\alpha_2 - \beta_2)$ ) | .48 | .35 |
| Panel B: Incentivised test | | |
| Levels $\times$ Teachers not incentivized in previous RCT ( $\alpha_1$ ) | .12 | .2*** |
| 1 (1) | (.072) | (.066) |
| Levels $\times$ Teachers incentivized in previous RCT ( $\alpha_2$ ) | .28*** | .29*** |
| 1 (2) | (.077) | (.087) |
| P4Pctile $\times$ Teachers not incentivized in previous RCT ( $\beta_1$ ) | 031 | .08 |
| 1 (17) | (.065) | (.068) |
| P4Pctile × Teachers incentivized in previous RCT ( $\beta_2$ ) | .17** | .17** |
| 1 (12) | (.073) | (.079) |
| Teachers incentivized in previous RCT | 087 | .088 |
| 1 | (.066) | (.085) |
| N. of obs. | 48,077 | 59,680 |
| $p$ -value( $H_0: \alpha_1 = \alpha_2$ ) | .048 | .36 |
| $p\text{-value}(H_0:\beta_1=\beta_2)$ | .02 | .34 |
| $p$ -value( $H_0: \alpha_1 = \beta_1$ ) | .054 | .088 |
| $p-value(H_0: \alpha_2 = \beta_2)$ | .15 | .11 |
| p-value( $H_0$ : $(\alpha_1 - \beta_1) = (\alpha_2 - \beta_2)$ ) | .59 | 1 |
The outcome is the composite test scores across math and Kiswahili. "Teachers incentivised in previous RCT" is equal to one if at least half of the teachers currently teaching Kiswahili or Math were eligible for incentives in the previous experiment (analysed by Mbiti *et al.* (2019)) in a treatment school. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
# **5 Cost-effectiveness**
We use accounting records to examine the cost-effectiveness of our interventions, following the framework outlined in [Dhaliwal](#page-49-9) *et al.* [\(2013\)](#page-49-9). The total annual cost of the teacher incentive programs was US\$7.23 per student.[48](#page-43-0) This cost estimate includes both the direct costs (value of incentive payments) and the implementation costs (test design and implementation, communications, audit, transfer costs, etc.) of the program. However, in the long run, the cost of the Pay for Percentile scheme is US\$1.50 higher (US\$8.73 total) due to pre-testing costs to determine ability groups.[49](#page-43-1)
We use the treatment effect on the composite index in the incentivised test in the second year to compute cost-effectiveness. We focus on the incentivised test to facilitate comparability with other teacher incentive studies. Since the Pay for Percentile treatment effect is 0.13*σ* in the second year, the cost-effectiveness of the intervention is 0.0148*σ* per dollar spent (assuming a linear dose-response relationship, for comparability with other studies, yields a cost-effectiveness of 1.48*σ* per US\$100 spent). On the other hand, the Levels treatment effect is 0.22*σ*, implying a cost-effectiveness of 0.0304*σ* per dollar spent (again, assuming a linear dose-response relationship for comparability with other studies yields a cost-effectiveness of 3.04*σ* per US\$100 spent). These estimates suggest that the Levels treatment is as effective as several other interventions in developing countries analysed in the overview by [Kremer](#page-50-8) *et al.* [\(2013\)](#page-50-8). For instance, the Levels treatment is more cost-effective than a computer-assisted learning program evaluated in India (1.54*σ* per US\$100) and the incentive program on attendance in India (2.28*σ* per US\$100).
# **6 Conclusion**
We use a randomised controlled trial to compare the effectiveness of two different teacher incentive programs to improve early-grade learning in Tanzanian public schools. Specifically, we compare the effectiveness of an innovative multiple-threshold proficiency incentive design relative to a more sophisticated
48For reference, average per-student expenditure, including teacher salaries, was ∼98 USD per student in 2014.
49The costs of pre-treatment testing required in Pay for Percentile for Grades 2 and 3 are not included in the cost figure since this cost would only be incurred once (ability groups could be based on endline data after the first year of implementation). Our calculations also assume similar data management costs for both programs, even though, in reality, the Pay for Percentile data costs were higher due to tasks such as preparing the ability groups and programming the payment calculations. However, these are primarily fixed costs and relatively small compared to the variable costs, especially at scale.
rank-order tournament-style Pay for Percentile system in terms of their impact on student test scores two years after the start of the program.
We report two sets of findings. First, both programs increase test scores compared to students in the control group. Moreover, the programs did not lead to negative learning spillovers in non-incentivised grades or subjects. Since neither program had any pedagogy or teacher training element, the learning effects are solely attributable to introducing teacher incentives. Both programs were equally effective for teachers with high and low content knowledge, and both programs benefited students across the skill distribution.
Second, despite the theoretical advantage of the Pay for Percentile system, our multiple-threshold proficiency system was at least as effective at increasing test scores and reducing grade repetition as the Pay for Percentile system. Combining these findings on program effectiveness with the lower cost of the Levels program, we conclude that Levels is the more cost-effective incentive program (although both programs are relatively cost-effective).
Our results demonstrate some theoretical and practical considerations education authorities interested in adopting teacher incentive programs must face. Although rank-order tournament schemes can provide powerful incentives to increase effort, such systems can be more opaque, making it harder for teachers to determine how to best exert effort. In contrast, the multiple-threshold proficiency system used in this study communicates clear student-level targets. These salient targets provide teachers with clear signals about how to allocate their effort in the class. Since developing countries often face implementation capacity constraints, the multiple-threshold system may be particularly well suited for these contexts, given its relative administrative simplicity. Further, such a system is arguably better suited for early grades, where the curriculum is focused on a narrower set of key learning milestones such as number recognition and subtraction. Consequently, this incentive system can serve as an important complement to "teaching at the right level" programs and education reforms that scale back overly ambitious curricula in early grades [\(Cunningham,](#page-48-9) [2018\)](#page-48-9).
An important caveat is that our results focus on short-run outcomes. In the long run, concerns about gaming the system (e.g., teaching to the test or cheating) will increase. Since rank-order tournaments (such as Pay for Percentile) allow education systems to use different tests and test formats, they can minimise these concerns if administrators are willing and able to implement such testing changes. Longer studies conducted at scale will be needed to understand better the long-run advantages and disadvantages of different teacher incentive systems.[50](#page-45-0)
# **Acknowledgments**
Twaweza funded this evaluation with supplemental funding from the REACH Trust Fund at the World Bank. We are grateful to Peter Holland, Jessica Lee, Arun Joshi, Owen Ozier, Salman Asim, and Cornelia Jesse for their support through the REACH initiative. Financial support from the Asociacion Mexicana ´ de Cultura, A.C. is gratefully acknowledged by Romero. This paper was partly written while Mbiti was visiting the Brown University Economics Department and the Population Studies and Training Center (PSTC).
# **Affiliations**
1Frank Batten School of Leadership and Public Policy, University of Virginia, 235 McCormick Road, Charlottesville, VA 22904
2 J-PAL
3Centro de Investigacion Econ ´ omica, ITAM, Av. Camino a Santa Teresa 930, CDMX, Mexico ´
4Twaweza, 15 Uganda Avenue, Dar Es Salaam, Tanzania
5NBER
50While some studies (e.g., [Lavy](#page-50-9) [\(2020\)](#page-50-9)) study long-term outcomes of being exposed to a teacher performance program, we refer to studies that focus on the steady-state outcomes after the program has been in place for several years (e.g., [Neal and Schanzenbach](#page-52-7) [\(2010\)](#page-52-7)).
6IZA
# **References**
- Ahrens, A., Hansen, C.B. and Schaffer, M.E. (2018). 'PDSLASSO: Stata module for post-selection and postregularization OLS or IV estimation and inference', Statistical Software Components, Boston College Department of Economics.
- Alger, V.E. (2014). 'Teacher incentive pay that works: A global survey of programs that improve student achievement', .
- Barlevy, G. and Neal, D. (2012). 'Pay for percentile', *American Economic Review*, vol. 102(5), pp. 1805–31, doi:10.1257/aer.102.5.1805.
- Barrera-Osorio, F., Cilliers, J., Cloutier, M.H. and Filmer, D. (2022). 'Heterogenous teacher effects of two incentive schemes: Evidence from a low-income country', *Journal of Development Economics*, vol. 156, p. 102820, ISSN 0304-3878, doi:https://doi.org/10.1016/j.jdeveco.2022.102820.
- Barrera-Osorio, F. and Raju, D. (2017). 'Teacher performance pay: Experimental evidence from Pakistan', *Journal of Public Economics*, vol. 148, pp. 75 – 91.
- Behrman, J.R., Parker, S.W., Todd, P.E. and Wolpin, K.I. (2015). 'Aligning learning incentives of students and teachers: Results from a social experiment in Mexican high schools', *Journal of Political Economy*, vol. 123(2), pp. 325–364, doi:10.1086/675910.
- Bettinger, E.P. and Long, B.T. (2010). 'Does cheaper mean better? the impact of using adjunct instructors on student outcomes', *The Review of Economics and Statistics*, vol. 92(3), pp. 598–613.
- Bietenbeck, J., Piopiunik, M. and Wiederhold, S. (2018). 'Africa's skill tragedy: Does teachers' lack of knowledge lead to low student performance?', *Journal of Human Resources*, vol. 53(3), pp. 553–578.
- Bold, T., Filmer, D., Martin, G., Molina, E., Stacy, B., Rockmore, C., Svensson, J. and Wane, W. (2017). 'Enrollment without learning: Teacher effort, knowledge, and skill in primary schools in Africa', *Journal of Economic Perspectives*, vol. 31(4), pp. 185–204.
- Breeding, M.E., Beteille, T. and Evans, D.K. (2021). 'Teacher pay-for-performance : What works? where? and how?', .
- Brehm, M., Imberman, S.A. and Lovenheim, M.F. (2017). 'Achievement effects of individual performance incentives in a teacher merit pay tournament', *Labour Economics*, vol. 44, pp. 133–150.
- Brown, C. and Andrabi, T. (2020). 'Inducing positive sorting through performance pay: Experimental evidence from pakistani schools', *University of California at Berkeley Working Paper*.
- Bruns, B., Filmer, D. and Patrinos, H.A. (2011). *Making schools work: New evidence on accountability reforms*, World Bank Publications.
- Charness, G. and Kuhn, P. (2011). 'Chapter 3 - lab labor: What can labor economists learn from the lab?', in (O. Ashenfelter and D. Card, eds.), *Handbook of Labor Economics*, pp. 229–330, vol. 4, Elsevier, doi:https://doi.org/10.1016/S0169-7218(11)00409-6.
- Chaudhury, N., Hammer, J., Kremer, M., Muralidharan, K. and Rogers, F.H. (2006). 'Missing in action: Teacher and health worker absence in developing countries', *Journal of Economic Perspectives*, vol. 20(1), pp. 91–116.
- Chetty, R., Friedman, J.N. and Rockoff, J.E. (2014a). 'Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates', *American Economic Review*, vol. 104(9), pp. 2593–2632.
- Chetty, R., Friedman, J.N. and Rockoff, J.E. (2014b). 'Measuring the impacts of teachers II: Teacher valueadded and student outcomes in adulthood', *American Economic Review*, vol. 104(9), pp. 2633–79.
- Contreras, D. and Rau, T. (2012). 'Tournament incentives for teachers: evidence from a scaled-up intervention in Chile', *Economic development and cultural change*, vol. 61(1), pp. 219–246.
- Cunningham, R. (2018). 'Unicef think piece series: Curriculum reform', UNICEF Eastern and Southern Africa Regional Office, Nairobi.
- Dalton, P.S., Gonzalez, V. and Noussair, C.N. (2015). 'Paying with self-chosen goals: Incentives and gender differences', CentER Discussion Paper Series No. 2016-036.
de Ree, J., Muralidharan, K., Pradhan, M. and Rogers, H. (2018). 'Double for nothing? experimental evidence on an unconditional teacher salary increase in Indonesia', *The Quarterly Journal of Economics*, vol. 133(2), pp. 993–1039.
- Dhaliwal, I., Duflo, E., Glennerster, R. and Tulloch, C. (2013). 'Comparative cost-effectiveness analysis to inform policy in developing countries: a general framework with applications for education', *Education Policy in Developing Countries*, pp. 285–338.
- Fehr, E., Klein, A. and Schmidt, K.M. (2007). 'Fairness and contract design', *Econometrica*, vol. 75(1), pp. 121–154, doi:https://doi.org/10.1111/j.1468-0262.2007.00734.x.
- Ferraz, C. and Bruns, B. (2012). 'Paying teachers to perform: The impact of bonus pay in Pernambuco, Brazil.', *Society for Research on Educational Effectiveness*.
- Figlio, D. and Loeb, S. (2011). 'Chapter 8 - school accountability', in (E. A. Hanushek, S. Machin and L. Woessmann, eds.), *Handbook of the Economics of Education*, pp. 383–421, vol. 3, Elsevier, doi:https: //doi.org/10.1016/B978-0-444-53429-3.00008-9.
- Fryer, R.G. (2013). 'Teacher incentives and student achievement: Evidence from New York City public schools', *Journal of Labor Economics*, vol. 31(2), pp. 373–407.
- Fryer, R.G., Levitt, S.D., List, J. and Sadoff, S. (Forthcoming). 'Enhancing the efficacy of teacher incentives through framing: A field experiment', *American Economic Journal: Economic Policy*.
- Gilligan, D.O., Karachiwalla, N., Kasirye, I., Lucas, A.M. and Neal, D. (2019). 'Educator incentives and educational triage in rural primary schools', *Journal of Human Resources*.
- Glewwe, P., Ilias, N. and Kremer, M. (2010). 'Teacher incentives', *American Economic Journal: Applied Economics*, vol. 2(3), pp. 205–27, doi:10.1257/app.2.3.205.
- Gneezy, U., List, J.A., Livingston, J.A., Qin, X., Sadoff, S. and Xu, Y. (2019). 'Measuring success in education: The role of effort on the test itself', *American Economic Review: Insights*, vol. 1(3), pp. 291–308, doi:10.1257/aeri.20180633.
- Gomez-Minambres, J. (2012). 'Motivation through goal setting', ´ *Journal of Economic Psychology*, vol. 33(6), pp. 1223 – 1239, ISSN 0167-4870, doi:https://doi.org/10.1016/j.joep.2012.08.010.
- Goodman, S.F. and Turner, L.J. (2013). 'The Design of Teacher Incentive Pay and Educational Outcomes: Evidence from the New York City Bonus Program', *Journal of Labor Economics*, vol. 31(2), pp. 409 – 420.
- Hanushek, E.A. and Rivkin, S.G. (2012). 'The distribution of teacher quality and implications for policy', *Annual Review of Economics*, vol. 4(1), pp. 131–157.
- Imberman, S.A. (2015). 'How effective are financial incentives for teachers?', *IZA World of Labor*.
- Kane, T.J., Rockoff, J.E. and Staiger, D.O. (2008). 'What does certification tell us about teacher effectiveness? evidence from New York City', *Economics of Education Review*, vol. 27(6), pp. 615–631.
- Koch, A. and Nafziger, J. (2011). 'Self-regulation through goal setting', *The Scandinavian Journal of Economics*, vol. 113(1), pp. 212–227, doi:10.1111/j.1467-9442.2010.01641.x.
- Kremer, M., Brannen, C. and Glennerster, R. (2013). 'The challenge of education and learning in the developing world', *Science*, vol. 340(6130), pp. 297–300.
- Kremer, M., Miguel, E. and Thornton, R. (2009). 'Incentives to learn', *The Review of Economics and statistics*, vol. 3(91), pp. 437–456.
- Ladd, H.F. (1999). 'The Dallas school accountability and incentive program: An evaluation of its impacts on student outcomes', *Economics of Education Review*, vol. 18(1), pp. 1–16.
- Lavy, V. (2002). 'Evaluating the effect of teachers' group performance incentives on pupil achievement', *Journal of Political Economy*, vol. 110(6), pp. pp. 1286–1317, ISSN 00223808.
- Lavy, V. (2009). 'Performance pay and teachers' effort, productivity, and grading ethics', *American Economic Review*, vol. 99(5), pp. 1979–2011, doi:10.1257/aer.99.5.1979.
- Lavy, V. (2020). 'Teachers' pay for performance in the long-run: The dynamic pattern of treatment effects on students' educational and labour market outcomes in adulthood', *The Review of Economic Studies*, ISSN 0034-6527, doi:10.1093/restud/rdaa002, rdaa002.
Leaver, C., Ozier, O., Serneels, P. and Zeitlin, A. (2021). 'Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from rwandan primary schools', *American Economic Review*, vol. 111(7), pp. 2213–46, doi:10.1257/aer.20191972.
- Lee, D.S. (2009). 'Training, wages, and sample selection: Estimating sharp bounds on treatment effects', *The Review of Economic Studies*, vol. 76(3), pp. 1071–1102.
- Leigh, A. (2012). 'The economics and politics of teacher merit pay', *CESifo Economic Studies*, vol. 59(1), pp. 1–33.
- Levitt, S.D., List, J.A., Neckermann, S. and Sadoff, S. (2016). 'The behavioralist goes to school: Leveraging behavioral economics to improve educational performance', *American Economic Journal: Economic Policy*, vol. 8(4), pp. 183–219.
- Loyalka, P., Sylvia, S., Liu, C., Chu, J. and Shi, Y. (2019). 'Pay by design: Teacher performance pay design and the distribution of student achievement', *Journal of Labor Economics*, vol. 37(3), pp. 621–662, doi:10. 1086/702625.
- Macartney, H. (2016). 'The dynamic effects of educational accountability', *Journal of Labor Economics*, vol. 34(1), pp. 1–28.
- Macartney, H., McMillan, R. and Petronijevic, U. (2021). 'A quantitative framework for analyzing the distributional effects of incentive schemes', National Bureau of Economic Research, doi:10.3386/w28816.
- Manacorda, M. (2012). 'The cost of grade retention', *Review of Economics and Statistics*, vol. 94(2), pp. 596–606.
- Mbiti, I. (2016). 'The need for accountability in education in developing countries', *Journal of Economic Perspectives*, vol. 30(3), pp. 109–32.
- Mbiti, I., Muralidharan, K., Romero, M., Schipper, Y., Manda, C. and Rajani, R. (2019). 'Inputs, incentives, and complementarities in education: Experimental evidence from Tanzania', *The Quarterly Journal of Economics*, vol. 134(3), pp. 1627–1673, ISSN 0033-5533, doi:10.1093/qje/qjz010.
- Mbiti, I., Romero, M. and Schipper, Y. (2022). 'Replication data for "Designing Effective Teacher Performance Pay Programs: Experimental Evidence from Tanzania"', doi:10.5281/zenodo.7411312.
- Metzler, J. and Woessmann, L. (2012). 'The impact of teacher subject knowledge on student achievement: Evidence from within-teacher within-student variation', *Journal of Development Economics*, vol. 99(2), pp. 486–496.
- Miller, G. and Babiarz, K. (2014). 'Pay-for-performance incentives in low- and middle-income country health programs', in (A. J. Culyer, ed.), *Encyclopedia of Health Economics*, pp. 457 – 466, San Diego: Elsevier.
- Mohanan, M., Donato, K., Miller, G., Truskinovsky, Y. and Vera-Hernandez, M. (Forthcoming). 'Different ´ strokes for different folks: Experimental evidence on the effectiveness of input and output incentive contracts for health care providers with varying skills', *American Economic Journal: Applied Economics*.
- Muralidharan, K. (2017). 'Chapter 3 - field experiments in education in developing countries', in (A. V. Banerjee and E. Duflo, eds.), *Handbook of Economic Field Experiments*, pp. 323–385, vol. 2 of *Handbook of Economic Field Experiments*, North-Holland, doi:https://doi.org/10.1016/bs.hefe.2016.09.004.
- Muralidharan, K. and Niehaus, P. (2017). 'Experimentation at scale', *Journal of Economic Perspectives*, vol. 31(4), pp. 103–24.
- Muralidharan, K. and Sundararaman, V. (2010). 'The impact of diagnostic feedback to teachers on student learning: Experimental evidence from India', *Economic Journal*, vol. 120, pp. F187–F203.
- Muralidharan, K. and Sundararaman, V. (2011). 'Teacher performance pay: Experimental evidence from India', *Journal of Political Economy*, vol. 119(1), pp. pp. 39–77.
- Neal, D. (2011). 'Chapter 6 - the design of performance pay in education', in (E. A. Hanushek, S. Machin and L. Woessmann, eds.), *Handbook of The Economics of Education*, pp. 495–550, vol. 4 of *Handbook of the Economics of Education*, Elsevier, doi:https://doi.org/10.1016/B978-0-444-53444-6.00006-7.
- Neal, D. and Schanzenbach, D.W. (2010). 'Left behind by design: Proficiency counts and test-based accountability', *Review of Economics and Statistics*, vol. 92(2), pp. 263–283, ISSN 0034-6535.
Niederle, M. and Vesterlund, L. (2007). 'Do women shy away from competition? do men compete too much?', *The Quarterly Journal of Economics*, vol. 122(3), pp. 1067–1101.
Niederle, M. and Vesterlund, L. (2011). 'Gender and competition', *Annual Review of Economics*, vol. 3(1), pp. 601–630.
OECD (2017). 'Teachers' salaries (indicator)', doi:10.1787/f689fb91-en, data retrieved from [https://data.](https://data.oecd.org/eduresource/teachers-salaries.htm) [oecd.org/eduresource/teachers-salaries.htm.](https://data.oecd.org/eduresource/teachers-salaries.htm)
Pham, L.D., Nguyen, T.D. and Springer, M.G. (2021). 'Teacher merit pay: A meta-analysis', *American Educational Research Journal*, vol. 58(3), pp. 527–566.
PRI (2013). 'Tanzanian teachers learning education doesn't pay', .
Renmans, D., Holvoet, N., Orach, C.G. and Criel, B. (2016). 'Opening the 'black box' of performance-based financing in low- and lower middle-income countries: a review of the literature', *Health Policy and Planning*, vol. 31(9), pp. 1297–1309, ISSN 0268-1080, doi:10.1093/heapol/czw045.
Reuters (2012). 'Tanzanian teachers in strike over pay', .
Singh, P. and Masters, W.A. (2018). 'Performance bonuses in the public sector: Winner-take-all prizes versus proportional payments to reduce child malnutrition in India', *Journal of Development Economics*, ISSN 0304-3878, doi:https://doi.org/10.1016/j.jdeveco.2018.10.003.
Uwezo (2012). 'Are our children learning? annual learning assessment report 2011', Uwezo, accessed on 05-12-2014.
Uwezo (2013). 'Are our children learning? numeracy and literacy across East Africa', Uwezo, Nairobi, accessed on 05-12-2014.
van der Linden, W.J. and Hambleton, R.K. (2013). *Handbook of modern item response theory*, Springer Science & Business Media.
Vigdor, J. (2008). 'Teacher salary bonuses in North Carolina', .
Woessmann, L. (2011). 'Cross-country evidence on teacher performance pay', *Economics of Education Review*, vol. 30(3), pp. 404–418.
World Bank (2015). 'Service delivery indicators: Tanzania', The World Bank, Washington D.C.
World Bank (2017a). 'Data for development: An evaluation of world bank support for data and statistical capacity', .
World Bank (2017b). 'World development indicators', Data retrieved from, [https://data.worldbank.org/](https://data.worldbank.org/data-catalog/world-development-indicators) [data-catalog/world-development-indicators.](https://data.worldbank.org/data-catalog/world-development-indicators)
World Bank (2018). 'World development report 2018: Learning to realize education's promise', doi:10. 1596/978-1-4648-1096-1.
# A Additional Figures and Tables
## A.1 Test score distribution at baseline
(a) Kiswahili (b) Math
Alignment of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of the control of
Figure A.1: CDF of test scores at baseline across experimental groups
# A.2 Effects on test takers (in the incentivised test)
| | (1)
Year 1 | (2)
Year 2 |
|-------------------------------------------------------------------|----------------------|------------------------|
| Levels $(\alpha_1)$ | 0.02
(0.02) | 0.05***
(0.01) |
| P4Pctile ( $\alpha_2$ ) | -0.00
(0.02) | 0.03**
(0.01) |
| N. of obs.
Mean control group $\alpha_3 = \alpha_2 - \alpha_1$ | 540
0.78
-0.02 | 540
0.83
-0.03** |
Table A.1: Number of test takers, incentivised test
The independent variable is the proportion of test takers (number of test takers divided by the enrolment in each grade) of the incentivised exam. The unit of observation is the school-grade level. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
0.20
0.04
p-value( $\alpha_3 = 0$ )
# A.3 Test score results for English
Table A.2: Effect on English test scores for grade 3
| | (1) | (2) | | | | | |
|---------------------------------------------|--------------|------------------|--|--|--|--|--|
| | Year 1 | Year 2 | | | | | |
| Panel A: Non-incentivised test | | | | | | | |
| | English | English | | | | | |
| Levels $(\alpha_1)$ | .012 | .11 | | | | | |
| | (.086) | (.085) | | | | | |
| P4Pctile ( $\alpha_2$ ) | 049 | .19** | | | | | |
| | (.076) | (.081) | | | | | |
| N. of obs. | 1,532 | 1,533 | | | | | |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 061 | .08 | | | | | |
| p-value ( $H_0: \alpha_3 = 0$ ) | .43 | .29 | | | | | |
| Panel B: Incentivised t | est | | | | | | |
| Levels $(\beta_1)$ | .28*** | .28*** | | | | | |
| V =/ | (.066) | (.069) | | | | | |
| P4Pctile ( $\beta_2$ ) | .16*** | .23*** | | | | | |
| 4 | (.057) | (.055) | | | | | |
| N. of obs. | 46,018 | 15,458 | | | | | |
| $\alpha_3 = \alpha_2 - \alpha_1$ | 12* | 047 | | | | | |
| p-value ( $H_0: \alpha_3 = 0$ ) | .079 | .53 | | | | | |
| Panel C: Incentivised t | est – Non-ir | ncentivised test | | | | | |
| $\gamma_1 = \beta_1 - \alpha_1$ | .14 | .15 | | | | | |
| $p$ -value( $\gamma_1 = 0$ ) | .12 | .14 | | | | | |
| $\gamma_2 = \beta_2 - \alpha_2$ | .2 | .04 | | | | | |
| $p$ -value( $\gamma_2 = 0$ ) | .017 | .66 | | | | | |
| $\gamma_3 = \beta_3 - \alpha_3$ | .053 | 11 | | | | | |
| $p$ -value( $\gamma_3 = 0$ ) | .54 | .28 | | | | | |
| $\gamma_1 - \gamma_2$ | 053 | .11 | | | | | |
| $p\text{-value}(\gamma_1 - \gamma_2 = 0)$ | .54 | .28 | | | | | |
| $\gamma_1 - \gamma_3$ | .2 | .04 | | | | | |
| $p\text{-value}(\gamma_1 - \gamma_3 = 0)$ | .017 | .66 | | | | | |
| $\gamma_2 - \gamma_3$ | .25 | 074 | | | | | |
| $p\text{-value}(\gamma_2 - \gamma_3 = 0)$ | .076 | .65 | | | | | |
| p-value( $\gamma_1 = \gamma_2 = \gamma_3$ ) | .054 | .33 | | | | | |
Results from estimating Equation 2 for different subjects at both follow-ups. Panel A uses data from the non-incentivised test taken by a sample of students. Control variables include student characteristics (age, gender, grade, and lag test scores) and school characteristics (PTR, Infrastructure PCA index, a PCA index of how close the school is to different facilities, and an indicator for whether the school is single shift or not). Panel B uses data from the incentivised test taken by all students. Control variables include student characteristics (gender and grade) and school characteristics (PTR, Infrastructure PCA index, a PCA index of how close the school is to different facilities, and an indicator for whether the school is single shift or not). Panel C tests the difference between the treatment estimates in panels A and B. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
## A.4 Pass rates
Table A.3: Pass rates across all skill levels
| | (1) (2) (3)
Year 1 | | (4) (5) (6)
Year 2 | | | |
|--------------------------------|-----------------------|-----------|-----------------------|----------|-----------|---------|
| | Math | Kiswahili | English | Math | Kiswahili | English |
| Levels $(\beta_1)$ | .0358** | .0582*** | .0359*** | .0366*** | .0682*** | .0149** |
| | (.015) | (.02) | (.0092) | (.013) | (.016) | (.006) |
| P4Pctile ( $\beta_2$ ) | .0224* | .00739 | .0169** | .0331*** | .0227 | .0132** |
| | (.012) | (.018) | (.0075) | (.012) | (.017) | (.0056) |
| N. of obs. | 210,358 | 129,676 | 129,676 | 248,250 | 181,288 | 30,986 |
| Control mean | .58 | .5 | .041 | .58 | .5 | .041 |
| $\beta_3 = \beta_2 - \beta_1$ | 013 | 051** | 019** | 0035 | 046*** | 0018 |
| p-value ( $H_0: \beta_3 = 0$ ) | .36 | .014 | .043 | .77 | .0051 | .8 |
The independent variable is whether a student acquired a given skill as evidenced by performance on the incentivised test. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
Table A.4: Pass rates using levels thresholds in Kiswahili
| | Syllables | Words | Sentences | Paragraph | Story | Reading
Comprehension |
|--------------------------------|-----------|---------|-----------|-----------|--------|--------------------------|
| | (1) | (2) | (3) | (4) | (5) | (6) |
| Panel A: Year 1 | | | | | | |
| Levels $(\beta_1)$ | .064** | .059** | .071*** | .075*** | .038 | .024 |
| | (.026) | (.024) | (.023) | (.022) | (.024) | (.026) |
| P4Pctile ( $\beta_2$ ) | 0057 | .015 | .011 | .026 | 0099 | 0034 |
| | (.025) | (.022) | (.021) | (.02) | (.021) | (.022) |
| N. of obs. | 17,886 | 33,440 | 33,440 | 15,554 | 14,678 | 14,678 |
| Control mean | .4 | .59 | .5 | .37 | .52 | .56 |
| $\beta_3 = \beta_2 - \beta_1$ | 069*** | 044* | 06** | 049** | 048** | 027 |
| p-value ( $H_0: \beta_3 = 0$ ) | .0086 | .081 | .011 | .017 | .045 | .27 |
| Panel B: Year 2 | | | | | | |
| Levels $(\beta_1)$ | .09*** | .085*** | .08*** | .046** | .0032 | .053** |
| • | (.021) | (.02) | (.018) | (.019) | (.026) | (.021) |
| P4Pctile ( $\beta_2$ ) | .047** | .036* | .032* | 0089 | 027 | .012 |
| • | (.023) | (.02) | (.019) | (.02) | (.022) | (.019) |
| N. of obs. | 26,746 | 44,262 | 44,262 | 17,516 | 15,493 | 33,009 |
| Control mean | .3 | .6 | .48 | .43 | .61 | .56 |
| $\beta_3 = \beta_2 - \beta_1$ | 044** | 049*** | 048*** | 055*** | 03 | 041* |
| p-value ( $H_0: \beta_3 = 0$ ) | .027 | .0082 | .0058 | .0042 | .22 | .053 |
The independent variable is whether a student acquired a given skill as evidenced by performance on the incentivised test. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
Table A.5: Pass rates using levels thresholds in math
| | Counting | Numbers | Inequalities | Addition | Subtraction | Multiplication | Division |
|--------------------------------|----------|----------|--------------|----------|-------------|----------------|----------|
| | (1) | (2) | (3) | (4) | (5) | (6) | (7) |
| Panel A: Year 1 | | | | | | | |
| Levels $(\beta_1)$ | .0034 | .014 | .03** | .05** | .043** | .038** | .035* |
| | (.0091) | (.021) | (.014) | (.021) | (.02) | (.017) | (.018) |
| P4Pctile ( $\beta_2$ ) | .031*** | .031* | .033*** | .018 | .016 | .023 | .0095 |
| | (.0078) | (.018) | (.012) | (.018) | (.016) | (.016) | (.018) |
| N. of obs. | 17,886 | 17,886 | 33,440 | 48,118 | 48,118 | 30,232 | 14,678 |
| Control mean | .93 | .64 | .74 | .59 | .5 | .23 | .22 |
| $\beta_3 = \beta_2 - \beta_1$ | .028*** | .017 | .0027 | 033 | 027 | 015 | 026 |
| p-value ( $H_0: \beta_3 = 0$ ) | .0012 | .4 | .85 | .12 | .16 | .37 | .17 |
| Panel B: Year 2 | | | | | | | |
| Levels $(\beta_1)$ | .000686 | .0411** | .0265** | .0442** | .0462** | .0514*** | .0395** |
| • | (.0078) | (.019) | (.011) | (.019) | (.019) | (.014) | (.017) |
| P4Pctile ( $\beta_2$ ) | .0108 | .0595*** | .0388*** | .0394** | .026 | .0254** | .0223 |
| | (.0071) | (.017) | (.01) | (.017) | (.017) | (.013) | (.017) |
| N. of obs. | 26,746 | 26,746 | 44,262 | 59,755 | 59,755 | 15,493 | 15,493 |
| Control mean | .94 | .68 | .79 | .6 | .56 | .11 | .18 |
| $\beta_3 = \beta_2 - \beta_1$ | .01 | .018 | .012 | 0049 | 02 | 026 | 017 |
| p-value ( $H_0: \beta_3 = 0$ ) | .12 | .31 | .23 | .78 | .24 | .11 | .34 |
The independent variable is whether a student acquired a given skill as evidenced by performance on the incentivised test. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
Table A.6: Pass rates using levels thresholds in English
| | Syllables | Words | Sentences | Paragraph | Story | Reading |
|--------------------------------|-----------|--------|-----------|-----------|---------|-------------------|
| | (1) | (2) | (3) | (4) | (5) | Comprehension (6) |
| Panel A: Year 1 | | | | | | |
| Levels $(\beta_1)$ | .095*** | .05*** | .023*** | .015** | .0079* | .013* |
| • | (.021) | (.013) | (.0087) | (.0065) | (.0046) | (.0078) |
| P4Pctile ( $\beta_2$ ) | .036** | .028** | .0041 | .0073 | .0079* | .019*** |
| • | (.016) | (.011) | (.007) | (.0055) | (.0046) | (.0064) |
| N. of obs. | 17,886 | 33,440 | 33,440 | 15,554 | 14,678 | 14,678 |
| Control mean | .087 | .075 | .023 | .007 | .021 | .036 |
| $\beta_3 = \beta_2 - \beta_1$ | 059*** | 022* | 019** | 0073 | 00001 | .0057 |
| p-value ( $H_0: \beta_3 = 0$ ) | .0034 | .074 | .043 | .29 | 1 | .44 |
| Panel B: Year 2 | | | | | | |
| Levels $(\beta_1)$ | | | | | .0074 | .022** |
| • | | | | | (.0061) | (.0086) |
| P4Pctile ( $\beta_2$ ) | | | | | .012* | .02** |
| , | | | | | (.0068) | (.0079) |
| N. of obs. | 0 | 0 | 0 | 0 | 10,735 | 10,735 |
| Control mean | | | | | .017 | .025 |
| $\beta_3 = \beta_2 - \beta_1$ | | | | | .0048 | 0016 |
| p-value ( $H_0: \beta_3 = 0$ ) | • | • | • | | .5 | .88 |
The independent variable is whether a student acquired a given skill as evidenced by performance on the incentivised test. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
# A.5 Lee Bounds for the incentivised test
Table A.7: Lee bounds for the incentivised test
| | (1) | (2) | (3) | (4) | |
|----------------------------------|---------|-----------|---------|-----------|--|
| | Year 1 | | Year 2 | | |
| | Math | Kiswahili | Math | Kiswahili | |
| Levels $(\alpha_1)$ | 0.11** | 0.13*** | 0.14*** | 0.18*** | |
| | (0.05) | (0.05) | (0.04) | (0.05) | |
| P4Pctile ( $\alpha_2$ ) | 0.07* | 0.02 | 0.09** | 0.09* | |
| | (0.04) | (0.04) | (0.04) | (0.05) | |
| N. of obs. | 48,077 | 48,077 | 59,680 | 59,680 | |
| $\alpha_3 = \alpha_2 - \alpha_1$ | -0.047 | -0.11** | -0.044 | -0.093** | |
| p-value( $\alpha_3 = 0$ ) | 0.30 | 0.026 | 0.31 | 0.045 | |
| Lower 95% CI ( $\alpha_1$ ) | 0.00066 | 0.021 | -0.023 | 0.027 | |
| Higher 95% CI ( $\alpha_1$ ) | 0.23 | 0.25 | 0.32 | 0.35 | |
| Lower 95% CI ( $\alpha_2$ ) | -0.012 | -0.070 | 0.014 | -0.0032 | |
| Higher 95% CI ( $\alpha_2$ ) | 0.14 | 0.10 | 0.17 | 0.17 | |
| Lower 95% CI ( $\alpha_3$ ) | -0.16 | -0.24 | -0.22 | -0.27 | |
| Higher 95% CI ( $\alpha_3$ ) | 0.063 | 0.00099 | 0.11 | 0.057 | |
The independent variable is the standardised test score for different subjects. For each subject, we present Lee (2009) bounds for all the treatment estimates (i.e., trimming the left/right tail of the distribution in Levels and P4Pctile schools so that the proportion of test-takers is the same as the number in control schools). Specifically, after we trim the data, we present the minimum of the lowest values within the confidence intervals (from trimming the left/right tail) and the maximum of the highest values within the confidence intervals. Standard errors, clustered at the school level, are in parentheses. \* p < 0.10, \*\* p < 0.05, \*\*\* p < 0.01
# A.6 Incentivised and non-incentivised tests matched sample
Table A.8: Difference between students matched across the incentivised and the non-incentivised test
| (1)
Non-matched | (2)
Matched | (3) | | | | |
|--------------------|-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--|--|--|--|
| | | Difference | | | | |
| Voor 1 | | | | | | |
| | 8 22 | 0.98*** | | | | |
| | | (0.05) | | | | |
| , , | . , | 0.03) | | | | |
| | | | | | | |
| | | (0.01)
0.17*** | | | | |
| | | | | | | |
| ` ' | ` ' | $(0.04)$ $0.17^{***}$ | | | | |
| | | 0.2. | | | | |
| , , | ` ' | (0.04)
0.28*** | | | | |
| | | | | | | |
| | | (0.04) | | | | |
| | | -0.07*** | | | | |
| | ` , | (0.02) | | | | |
| | | -0.05*** | | | | |
| (0.46) | (0.48) | (0.02) | | | | |
| Panel B: Year 2 | | | | | | |
| 8.60 | 8.01 | 0.60*** | | | | |
| (1.37) | (1.23) | (0.05) | | | | |
| 0.51 | 0.49 | 0.02 | | | | |
| (0.50) | (0.50) | (0.01) | | | | |
| -0.05 | -0.32 | 0.26*** | | | | |
| (0.99) | (0.90) | (0.04) | | | | |
| -0.04 | -0.31 | 0.27*** | | | | |
| (1.01) | (0.79) | (0.04) | | | | |
| -0.07 | -0.42 | 0.34*** | | | | |
| | | (0.04) | | | | |
| 0.31 | 0.36 | -0.05** | | | | |
| | | (0.02) | | | | |
| | | -0.03 | | | | |
| | | (0.02) | | | | |
| | 8.60
(1.37)
0.51
(0.50)
-0.05
(0.99)
-0.04
(1.01)
-0.07
(0.99) | 9.29 8.32 (1.68) (1.33) 0.50 0.48 (0.50) (0.50) 0.10 -0.07 (1.02) (0.96) 0.09 -0.08 (1.08) (0.94) 0.13 -0.15 (1.05) (0.95) 0.31 0.37 (0.46) (0.48) 0.31 0.36 (0.46) (0.48) (1.37) (1.23) 0.51 0.49 (0.50) (0.50) -0.05 -0.32 (0.99) (0.90) -0.04 -0.31 (1.01) (0.79) -0.07 -0.42 (0.99) (0.86) 0.31 0.36 (0.46) (0.48) | | | | |
This table presents the mean and standard error of the mean (in parentheses) for several characteristics of students that we were able to match across the incentivised and the non-incentivised test. The match was manually via a fuzzy merge based on student's names. Column 3 shows the difference between the two groups (and the standard error of the difference). Standard errors are clustered at the school level for the test of equality. \* p < 0.10, \*\*\* p < 0.05, \*\*\*\*