Would it be outrageous for economists to have to pay back part of their salaries if their theories don’t improve the economy? What if education economists had to return consulting fees to the think tanks that paid them if their methods were flawed? Or if their conclusions ignored other research findings and the realities of teaching?
If you think this would be a good idea, then Harvard professor and MacArthur Genius Grant winner Roland Fryer and Freakonomics co-author and University of Chicago professor Steven Levitt (and colleagues John List and Sally Sadoff) might just have to cancel their vacations this fall. They would have to return the pay they received for their recent paper and recommendation that teachers must return salary bonuses (paid up front) when their students’ test scores do not improve sufficiently from one year to the next.
The Study: What It Said
Funded by the Broad Foundation—and drawing on a study about how Chinese factory workers are paid (see page 2)—the report concludes that teacher-incentive programs should be framed in terms of losses rather than gains.
Frustrated by study after study showing that single-minded merit-pay plans don’t improve student achievement (e.g., see the RAND report on New York City’s program, and the exhaustive longitudinal Nashville experiment), these highly touted scholars are on a mission to prove, as one journalist suggested, that “the promise of a larger paycheck will motivate (teachers) to work harder when they're up in front of the chalkboard.”
Fryer and colleagues analyzed the student test results (from the ThinkLink Assessment System) of 150 K-8 teachers from Chicago Heights, Illinois. The teachers were divvied into one control group and four treatment groups, with opportunities to be rewarded or punished for individual and team gains or losses. The “gain” treatment group worked under a traditional year-end bonus structure, similar to merit pay proposals of past and present. This group could earn up to $8,000 extra based on their students' standardized test scores.
Another treatment group—the “loss” treatment group—was given $4,000 at the beginning of the year and knew up front that if their students scored below expectations on the end-of-year test, they'd have to return part of this payment, proportionate to their students’ results. The researchers found that the gain group showed no significant differences in student achievement, while the loss-treatment group’s individual and team results were comparable to increasing teacher quality by one standard deviation.
What It Missed
I am certain that this study was conducted by very smart people, but people who know too little about teaching and learning, schooling and community, teachers and students.
Consider these issues:
1. The methodology is built on the assumption that money makes teachers more effective, but there is a wealth of evidence, including a recent Gates-funded study, to the contrary.
2. The researchers used a low-stakes standardized diagnostic test that is supposed to be aligned with state standards, but there is no evidence that is the case. Did the researchers read this recent report on how off the mark many standardized tests are?
3. While the teachers were assigned randomly to the control and treatment groups to control for working conditions, the researchers do not own up to the fact they are making a summative judgment about teacher effectiveness on the basis of a 25-50-question multiple-choice test that may not be aligned with the standards that teachers had been using all year.
The researchers did show some concern for how other factors might affect the achievement of these students. They looked at student mobility. Check. They look at cheating. Check. They look at liquidity constraints. (What?) Fryer and company considered how teachers who received upfront payments might have competitive advantage if they “purchase certain materials that facilitate instruction.” My first reaction was to offer kudos to Professor Fryer. Not so fast.
Fryer was concerned that teachers who had additional dollars up front might go out and buy “new workbooks or dry-erase markers” that would allow them to teach more effectively than those in the control and gain groups. Really? These concerns tell us so much about how little these researchers know about teaching and learning.
Other studies have shown that teachers are more effective when they loop with their students for more than one year, when they teach the same course (in secondary school) for more than one year, when they have high-quality curriculum materials, and when they work with a team to plan curricula and evaluate student work. All these factors contribute far more to effective teaching than the loss-aversion merit pay schemes posed by this study.
More questions arise about how Fryer’s scheme would work in reality: Would districts have to hire more bureaucrats to collect the dollars awarded to teachers when test scores did not increase? How would such a pay scheme affect teacher morale? What about the impact on collaboration, given that the authors’ statistical model promotes competition among teachers?
Economists like Fryer and company—and their funders—are desperate to find some kind of empirical link between short-term test score gains and teacher pay (or firings). The researchers found a model—based on a Chinese factory where “loss” workers saw only a 1 percent increase in productivity—and decided to use it in such a complex enterprise as teaching.
If they were truly interested in improving schools, they might want to take a look at teachers’ own bold ideas about pay. If they were truly interested in improving schools, they might want to take a look at a larger body of research on teacher incentives and student achievement. Perhaps in the future, economists like Fryer and company will have to return their consulting fees if they don’t.