Advancing the Teaching Profession

New Report: Use VAM Data Carefully to Assess Teachers

Why do so many journalists and policy wonks think it’s okay to use imperfect data as a lone measure to judge teachers?

I agree that perfection should not get in way of progress. But the LA Times (and a number of think tank analysts) continue to justify the use of VAM — value-added estimates of individual teachers — as the sole determinant of who is effective in raising student achievement and who is not.

The pundits may give a slight tip of the hat toward multiple measures of effectiveness, but their conventional wisdom (as surfaced again in an August 28th LA Times article) is that “the benefits of singling out those who consistently succeed” far exceeds the bad “publicity” for those with “low rankings” (and the lack of fully reliable data be damned).
 
While Secretary of Education Arne Duncan has said that “no one thinks test scores should be the only factor in teacher evaluations,” he and many others may be encouraging school districts to make it the overwhelming factor when they give their blessing for newspapers to identify effective teachers solely on the basis of VAM mathematical formulas, as the Secretary did with some gusto in his August 29, 2010 article for the New York Daily News.

Meanwhile the hard research is piling up, showing how the use of standardized student test scores and value-added modeling in teacher evaluation can do lots more harm than good — and harm not just to teachers, but to schools in every kind of American community, by distorting the reasons for student success.

A major new study from a top national team of education measurement experts* offers up mounds of evidence to confirm how unstable the results of VAM evaluations are across time, across classes and across tests. These respected scholars, assembled by the Economic Policy Institute, found that applying value-added methodologies to student test scores does not fully control for the varied factors that affect student learning, including the effects of summer learning loss for some students.

They also reveal how VAM does not account for the fact that some students receive instruction in particular subjects from more than one teacher or that some are offered afterschool programs, while others are not. And they make an even more critical point:

Creating a system in which teachers are, in effect, competing with each other can reduce the incentive to collaborate within schools – and studies have shown that better schools are marked by teaching staffs that work together.

The Secretary suggests that the issue of publicly releasing the names of effective and ineffective teachers in the newspaper, based solely on VAM results, is an “emotional” one. But I would argue that this issue is far more about science than emotion.

The science is clear: VAM stratagems — no matter how sophisticated their statistical procedures may be — are not capable alone of fairly and accurately identifying who is an effective teacher. There’s some irony in the fact that this major new study supports an analysis actually commissioned by the Secretary recently. The Secretary’s report concluded that over 1 in 4 teachers who are judged by VAM tools will be falsely labeled as effective or ineffective, even when there are three years worth of data available.
 
In its most recent story on teacher performance, the LA Times  identified and painted a powerful portrait of a number of effective teachers like Zenaida Tan and Hollie Bloch, two very experienced educators who teach in LAUSD. But the further irony is that while VAM helped the Times find these two teachers, there is at least one chance in four that Ms. Tan and Ms. Bloch might be labeled “ineffective” by the newspaper in some future year, using the same exact methods.

Returning to the EPI report, I could not agree more with America’s educational measurement experts that “although standardized test scores of students are one piece of information for school leaders to use to make judgments about teacher effectiveness, such scores should be only a part of an overall comprehensive evaluation.”

I agree with those who say that perfection should not get in the way of progress. A comprehensive system of evaluating teacher performance would include VAM measures as one component, but also an array of other metrics — e.g., teachers’ analyses of student work or how teachers spread expertise from one classroom to the next. What’s more, a really good system would reveal why some teachers were more effective than others and what we can do to increase their numbers and their spread.

Such a system also would rely on the leadership of classroom experts like Tan and Bloch, and not on school principals who have neither the time nor training to make the granular pedagogical decisions that reveal who is effective or not.

I agree with the Secretary that a public spotlight needs to shine on our nation’s most effective teachers. I’ve been advocating for just such practices for more than a decade. But let’s make sure we understand all the dimensions of teaching that make them outstanding. And then let's advocate — together — for a system that guarantees and rewards those expert practices.

____

* The distinguished authors of EPI's report include four former presidents of the American Educational Research Association; two  former presidents of the National Council on Measurement in Education;  the current and two  former chairs of the Board of Testing and Assessment of the National Research Council of the National Academy of Sciences;  the president-elect of the Association for Public Policy Analysis and Management;  the former director of the Educational Testing Service's Policy Information Center and a former associate director of the National Assessment of Educational Progress; a former assistant U.S. Secretary of Education; a former  and current member of the National Assessment Governing Board; and the current vice-president, a former president, and three other members of the National Academy of Education.

The School Reform Policy Gap

The latest PDK/Gallup Poll reveals a significant difference in the opinions of the American public and the Obama Administration over the best ways to reform schools and assure teaching quality.

In 2010, more than one in three Americans actually gave the President a “D” or a “F” for his performance in support of public schools — a much poorer showing than the year before. What has changed? Since the 2009 poll, the Administration has laid out the specifics of its plan to turn around low-performing schools and improve teacher performance – and to significantly increase the number of charter schools in the U.S.

The poll surfaced several disconnects between current policy and public sentiment:
 
• While the Administration wants to make it easier to recruit teachers to high needs schools through short-cut alternative certification programs, the American people believe teachers need “more time to learn to teach in new ways.”

• While the Administration has put emphasis on firing “bad” teachers (e.g., Central Falls High School in Rhode Island), the American people believe the call for mass firings of school faculty is wrong-headed. In the poll, 54% prefer a strategy specifically aimed at improving teaching skills. Only 13 percent think it’s a good idea to close a low-performing school and re-open it as a charter (a la New Orleans).
 
• On the question of school reform priorities, Americans believe that improving the quality of teachers is “more important” (44%) than measuring student achievement through accountability testing (11%) or developing more demanding education standards (24%).

• A sizable majority of Americans (71%) believe teachers’ pay should be tied to their work, not just on a “standard scale basis.” But they are also increasingly skeptical about using current standardized tests to judge students and teachers — and they are now much less likely to want teacher pay to be “very closely tied” to students’ academic achievement. In the 2000 PDK poll, 25 percent called for teacher pay to be tightly linked to student test scores. In 2010, that percentage dropped to 19.

As I told Education Week reporter Dakarai I. Aarons recently, there is far more interest among the American public in supporting teachers than in firing them or paying them on the basis of test scores. That doesn’t mean the American people don’t want a results-oriented profession. They do. But they seem to understand (better than many policymakers) that most teachers who are not effective can become so.

Indeed, the 2010 poll suggests that Americans have quite a bit of faith in their public school teachers.

Far more public school parents today, compared to a decade ago, believe schools have “caused” their children to be “eager” about learning (63% in 2010, compared to 50% in 1998). What’s more, over 70 percent of Americans (not just public school parents) “have trust and confidence” in those who teach children in the public schools.

Far too many policy approaches today view teachers as the problem to be fixed — not as professionals who can create and enact solutions with the right supports. Could this negative outlook on teaching explain why more Americans are giving the President failing grades for his performance in supporting public education?

Teachers and Valued Added Tools: Finding the Right Way, Far from the Madding Crowd

Saturday’s Los Angeles Times focuses once again on the value-added data it has assembled to judge instructional performance in the LAUSD system. But the emphasis now is more on schools than teachers. The piece, penned by Jason Song, Jason Felch and Doug Smith, tunes in on which schools produce higher value-added gains than others — and why (well, only “kinda” on the why). The authors avoided putting too much of a spotlight on the newspaper’s earlier decision to use this same data set to identify publicly which teachers are THE effective ones and which ones are NOT.
 
Since last week’s LA Times article hit the press, some of the nation’s experts on value-added methodology (VAM), like Doug Harris, and advocates for their use, like Rick Hess, have challenged the newspaper's decision to publish individual teachers’ ratings. Both are quoted in Carl Bialik’s thoughtful critique in the Wall Street Journal, where he concluded that the metric being used is way too unstable to give consistently accurate results of which individual teachers are more effective than others.

Even Dan Goldhaber, a well-known labor economist who routinely develops and uses these tools in education contexts, “strongly opposes” making individual teachers' value-added scores public in the way done by the LA Times. In recent comments on the Web, Goldhaber noted that there are “legitimate reasons” — like pull out programs and migration of students in and out of individual teachers’ classrooms (which is very common in LA) — that “make the attribution of student (achievement) to (specific) teachers complex.”

The VAM tool can be very useful in helping teachers get better at what they do, as also suggested by the new Song, et. al. LA Times article. And the district and the union in Los Angeles should find a way to use value-added data to help teachers improve their teaching practice and spread their expertise. However, as an isolated metric, VAM is not a fair method (as some policy pundits and policy zealots suggest) to praise or condemn individual teachers.

Reformers have long yearned for tools to turn teaching into a results-oriented profession. I am one of them. But the growing enthusiasm for using value-added formulas to offer summative judgments on the effectiveness of individual teachers is unwarranted, when research shows that 25-30% of those identified as effective in one year would be identified as ineffective the next.

The policy and media voices advocating for the indiscriminate use of VAM tools remind me a bit of “the madding crowd's ignoble strife” described by 18th century English poet Thomas Gray. And I also recall the comment of Scottish historian Charles Mackay, who once observed:

Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, one by one.

Teaching Effectiveness and the LA Times: A Debacle in the Making

In an interview yesterday with Amanda Paulson of the Christian Science Monitor, I told her that the decision by the Los Angeles Times to publish a list of effective and ineffective teachers, based solely on the results of one researcher’s take on value-added estimates, was absolutely “god-awful.”

Don’t get me wrong: As an organization we are dedicated to advancing a results-oriented profession, which means teacher evaluation metrics must include evidence of student learning. But the approach used by the L.A. Times is rife with huge problems — most notably the fact that value-added data on teacher effectiveness is far too unstable to use a sole arbiter of who is an effective teacher and who is not. Even the U.S. Department of Education, which is promoting the use of value-added data in teaching quality reforms, released a major research paper last month pointing to the substantial error rates associated with generating lists of good and not-so-good teachers using these methods.

This isn’t an ideological issue. Rick Hess of the American Enterprise Institute and I are frequently at odds about the best ways to assure teaching quality now and in the future. But we are in agreement on the poor judgment underlying the L.A. Times decision. Hess noted in his Education Week blog this morning that he is “increasingly nervous at how casually reading and math value-added calculations are being treated as de facto determinants of ‘good’ teaching.” And well he should be.

Elsewhere I have pointed to a lengthy list of inherent problems in using VAM to identify effective teachers — including the following realities:

1. Students are not randomly assigned to teachers, and VAM measures cannot fully sort out the effects caused by differences in students’ needs or preparedness — as distinct from teacher effects.

2. The lack of properly scaled year-to-year tests makes it difficult to evaluate gains at all points along the achievement continuum or to assess (for example) a physics teacher’s effectiveness based on her students’ previous scores in chemistry.

3. Many students in high-needs schools are highly mobile and do not complete a full year of instruction in a given teacher’s classroom.

4. Many students are taught the same subjects by more than one teacher — confounding the capacity of distant statistical models to isolate the effects from individual classroom instructors.
 
5. In addition, depending on the VAM statistical model a researcher uses, the same teacher can be identified as effective or ineffective.

6. Finally, researchers have found that the same teacher’s effectiveness rating can change depending on the school in which he or she teaches.

No wonder the USDOE report revealed that even when three years of VAM data are used, over 25 percent of the teachers will be “erroneously identified.” That should make any fair-minded person think twice about using such data to name names and possibly destroy careers.

The Bill & Melinda Gates Foundation, under the auspices of its Measuring Effective Teaching project, is taking a very thoughtful approach to teacher assessment by looking at multiple measures of student achievement and linking other metrics (e.g., classroom observations, teachers’ analyses of student work and their own teaching, and levels of student engagement) to capture a more robust and accurate view of who is effective and why.

If policymakers and the media mavens really want to learn about effective teacher evaluation, it’s time they turn to the wisdom of those who actually teach — like Larry Ferlazzo, a Teacher Leaders Network member, who penned a thoughtful response in this morning’s Washington Post blog The Answer Sheet to what will surely be a journalistic debacle.

I suspect the Los Angeles Times will soon hear from many teachers branded as “ineffective” who turn out to be “false negatives” (or a Type II error in statistical lingo), and the Times editors will need to re-think their approach to reporting on the debates about effective teaching.

I’m frankly startled by the Times's decision to publish this highly questionable data and risk humiliating many competent teachers by doing so. I wonder how journalists of national stature, who are trained to look at multiple sides of an issue and to be sure of their facts, could be so reckless?

Test Score Gains Fade in NYC and DC: Time to Focus on Teaching Quality?

News of test score “fade” from both New York City and the District of Columbia schools is surfacing. Both districts — often held up as emerging models of school reform — have used a combination of narrowed curriculum, lock-step teaching procedures, and high stakes teacher accountability to produce some short-term gains. But now the story lines of Joel Klein and Michelle Rhee, two of the nation’s most well-known school superintendents, may need to change.

With the release of 2010 results, NYC passing rates dropped by more than 25 percent and the achievement gap (between white and minority students) widened substantially. In DC, school leaders had to admit that their heralded gains in the elementary schools over two previous years are now evaporating. They do not know why.

Test score gain can fluctuate for a lot of reasons, including statistical error and student mobility. and for these reasons (and others) experts claim that no test should be used as the primary arbiter of who is doing the best job of educating children.

Dan Koretz, one of the nation’s leading experts on student assessment, has concluded (p. 8) that high stakes testing often results in "substantial distortions of practice" in the classroom "and inflation of test scores." He writes: "The seriousness of this problem is hard to overstate. When scores are inflated, many of the most important conclusions people base on them will be wrong, and students -- and sometimes teachers -- will suffer as a result."

But well-documented observations by Koretz and others have not deterred zealous school reformers or journalists looking to make a splashy headline, such as Newsweek’s recent in-your-face cover story on “bad teachers” as the root cause of America’s failing public education system.

I don’t know any supporters of a positive approach to teaching quality and teacher evaluation who would question whether NYC or DC schools need to overhaul how they recruit, support, assess and pay teachers. That’s clearly what’s best for students. But I do wonder if the recent failings of Klein and Rhee’s heralded reform schemes — which grow out of a one-dimensional view of effectiveness — will raise more questions among the public about how we judge schools and the teachers who teach in them.

Maybe we’re moving toward a tipping point. I’m encouraged that Valerie Strauss of the Washington Post has opened up her blog The Answer Sheet to voices from many sides of the assessment issue — and especially to teacher voices. I wonder if there may be more of an interest now in how to elevate and spread the performance of effective teachers who do more than raise a test score a point or two — and ultimately less interest in firing a few teachers who do not always generate gains in lock-step ways from year to year.  

I’m feeling a bit optimistic: I'm sensing that more and more of us want to focus on effective teaching and how to cultivate, spread and sustain it — and not on a paradigm of punishment and recrimination.

Big Ideas from Some of Our Nation’s Most Effective (and potentially most influential) Teachers

This past weekend I had the pleasure of working closely with 19 extraordinary teachers from throughout the United States – a sample of delegates drawn from the hundreds of accomplished educators who participate in the many initiatives of our Teacher Leaders Network.

We worked hard and played hard as we gathered for a three-day retreat at CTQ headquarters in Hillsborough, North Carolina. We dug deeply into what it means to transform teaching into a 21st century profession fully prepared to help every student succeed in the decades to come, both in brick-and-mortar schools and in cyberspace.

I learned so much from Tom Bremer (Illinois), Isabel Campos-Woytek (Colorado), Ben Crosby (California), Julianna Dauble (Washington), Nina Davis (North Carolina), Susan Graham (Virginia), Sarah Henchey (North Carolina), John M. Holland (Virginia), Marguerite Izzo (New York), Kris Kohl (Nevada), Patrick Ledesma (Virginia), Lori Nazareno (Colorado), Ernie Rambo (Nevada), Marsha Ratzel (Kansas), Cindi Rigsbee (North Carolina), Zach Rupp (Colorado), Ariel Sacks (New York City), Bob Williams (Alaska), and Heather Wolpert-Gawron (California). (We really missed Renee Moore from Mississippi who could not attend at the last minute.)

Teachers in this amazing group fully represent the diversity of America’s teaching force – teaching a variety of subjects and fulfilling many leadership roles in urban, suburban and rural schools at every level of the PK-12 spectrum. They are involved in our Teacher Leaders Network Forum; our emerging network of state and national teachers of the year (thanks to CCSSO); our book-writing team which has been designing a powerful pathway toward Teaching 2030;  our TeacherSolutions group studying the kinds of conditions teachers need in order to teach effectively in high needs schools. Five of them are part of our New Millennium Initiatives in four states. Several are Teach for America alums, or other first- and second-career teachers who entered the profession through alternative certifications. Others graduated from traditional schools of education.

They are all committed to improving preparation before anyone teaches anyone’s child. They have strategies for identifying and spreading the expertise of effective teachers. They can show you how to assess students for 21st century knowledge and skills. They came up with big and break-through policy proposals for how to do so. They have passion and knowledge as well as pedagogical expertise that rarely exists inside the D.C. Beltway, where most education policy gets debated and crafted. They transcend traditional union and administrator policy positions as well as those of left-wing and right-wing think tanks. If policymakers want to know how to get to “and/both” teaching and education policy -- and leave “either/or” approaches in the dust — ask these teacher leaders.

Thanks to several foundations supporting projects these energized educators are engaged in — including Ford, Bill & Melinda Gates, Joyce, MetLife, Rose and Stuart — you will hear more from these extraordinary teachers in resounding and systematic ways. Since fleeing the classroom in 1981 to work on the transformation of teaching into the profession students deserve, I have never been any more inspired than I am by our work this past weekend. I have never been so hopeful for the possibilities that lie ahead. Stay tuned.

The Highest Paid Anybody

I have said for some time that I will continue to advocate for the teaching profession until the highest paid anybody in a school system is a practicing teacher. Think about the implications: For this idea to become a reality our education system would have to not only identify excellent teachers, but have a variety of venues to make them more visible to policymakers and the public as well as their fellow practitioners.

These top-paid educators would continue to teach part of the day, week, or year, but also spread their expertise in a variety of ways — perhaps as Teacherpreneurs (a concept outlined in Teaching 2030, a book I've penned with 12 of my teacher colleagues from the Teacher Leaders Network, with generous support from MetLife Foundation.)

While teachers and parents cheer when they hear me offer up this possibility, many of the D.C. policy pundits roll their eyes and quietly scoff at the idea. Well now a group of economists have confirmed that teachers are worth quite a bit — perhaps more than many of our top-paid urban superintendents.
 
Raj Chetty and colleagues found that even very young students who had more effective teachers and learned more in kindergarten were more likely to go college and earn more (and in doing so) produce more economic benefit to society. An effective kindergarten teacher, according to the economists, is worth $320,000. The researchers found that the effects of effective teachers were not found in year-to-year gains on standardized tests but in long-term adult outcomes. (They study drew upon a database of 12,000 children who were part of a well-designed scientific investigation in Tennessee in the 1980s and are now in their 30s.)

A recent New York Times article calls the findings “explosive.” I would suggest they are re-affirming our work at the Center for Teaching Quality — and should provoke a new narrative on how policymakers frame efforts to recruit and reward effective teachers.

• We need to think and act well beyond the narrow (and narrow-minded) strategy of focusing on ridding schools of a few ineffective teachers or paying teachers for short-term gains determined on 20th century standardized tests.

• We need to begin to blur the lines of distinction between those who lead schools and those who teach in them.

• We need to place a premium on those who produce long-term results and spread their expertise to others.

We need to really value excellent early childhood teachers, like my Teaching 2030 colleague and co-author John Holland. I will be the first to say that John — a 12-year veteran of teaching three and four-year-olds from Richmond’s toughest neighborhoods who is also an artist and innovator and thoughtful blogger for Inside Pre-K — is worth $320,000 a year.