In an interview yesterday with Amanda Paulson of the Christian Science Monitor, I told her that the decision by the Los Angeles Times to publish a list of effective and ineffective teachers, based solely on the results of one researcher’s take on value-added estimates, was absolutely “god-awful.”
Don’t get me wrong: As an organization we are dedicated to advancing a results-oriented profession, which means teacher evaluation metrics must include evidence of student learning. But the approach used by the L.A. Times is rife with huge problems — most notably the fact that value-added data on teacher effectiveness is far too unstable to use a sole arbiter of who is an effective teacher and who is not. Even the U.S. Department of Education, which is promoting the use of value-added data in teaching quality reforms, released a major research paper last month pointing to the substantial error rates associated with generating lists of good and not-so-good teachers using these methods.
This isn’t an ideological issue. Rick Hess of the American Enterprise Institute and I are frequently at odds about the best ways to assure teaching quality now and in the future. But we are in agreement on the poor judgment underlying the L.A. Times decision. Hess noted in his Education Week blog this morning that he is “increasingly nervous at how casually reading and math value-added calculations are being treated as de facto determinants of ‘good’ teaching.” And well he should be.
Elsewhere I have pointed to a lengthy list of inherent problems in using VAM to identify effective teachers — including the following realities:
The Bill & Melinda Gates Foundation, under the auspices of its Measuring Effective Teaching project, is taking a very thoughtful approach to teacher assessment by looking at multiple measures of student achievement and linking other metrics (e.g., classroom observations, teachers’ analyses of student work and their own teaching, and levels of student engagement) to capture a more robust and accurate view of who is effective and why.
If policymakers and the media mavens really want to learn about effective teacher evaluation, it’s time they turn to the wisdom of those who actually teach — like Larry Ferlazzo, a Teacher Leaders Network member, who penned a thoughtful response in this morning’s Washington Post blog The Answer Sheet to what will surely be a journalistic debacle.
I suspect the Los Angeles Times will soon hear from many teachers branded as “ineffective” who turn out to be “false negatives” (or a Type II error in statistical lingo), and the Times editors will need to re-think their approach to reporting on the debates about effective teaching.
I’m frankly startled by the Times's decision to publish this highly questionable data and risk humiliating many competent teachers by doing so. I wonder how journalists of national stature, who are trained to look at multiple sides of an issue and to be sure of their facts, could be so reckless?
Don’t get me wrong: As an organization we are dedicated to advancing a results-oriented profession, which means teacher evaluation metrics must include evidence of student learning. But the approach used by the L.A. Times is rife with huge problems — most notably the fact that value-added data on teacher effectiveness is far too unstable to use a sole arbiter of who is an effective teacher and who is not. Even the U.S. Department of Education, which is promoting the use of value-added data in teaching quality reforms, released a major research paper last month pointing to the substantial error rates associated with generating lists of good and not-so-good teachers using these methods.
This isn’t an ideological issue. Rick Hess of the American Enterprise Institute and I are frequently at odds about the best ways to assure teaching quality now and in the future. But we are in agreement on the poor judgment underlying the L.A. Times decision. Hess noted in his Education Week blog this morning that he is “increasingly nervous at how casually reading and math value-added calculations are being treated as de facto determinants of ‘good’ teaching.” And well he should be.
Elsewhere I have pointed to a lengthy list of inherent problems in using VAM to identify effective teachers — including the following realities:
1. Students are not randomly assigned to teachers, and VAM measures cannot fully sort out the effects caused by differences in students’ needs or preparedness — as distinct from teacher effects.No wonder the USDOE report revealed that even when three years of VAM data are used, over 25 percent of the teachers will be “erroneously identified.” That should make any fair-minded person think twice about using such data to name names and possibly destroy careers.
2. The lack of properly scaled year-to-year tests makes it difficult to evaluate gains at all points along the achievement continuum or to assess (for example) a physics teacher’s effectiveness based on her students’ previous scores in chemistry.
3. Many students in high-needs schools are highly mobile and do not complete a full year of instruction in a given teacher’s classroom.
4. Many students are taught the same subjects by more than one teacher — confounding the capacity of distant statistical models to isolate the effects from individual classroom instructors.
5. In addition, depending on the VAM statistical model a researcher uses, the same teacher can be identified as effective or ineffective.
6. Finally, researchers have found that the same teacher’s effectiveness rating can change depending on the school in which he or she teaches.
The Bill & Melinda Gates Foundation, under the auspices of its Measuring Effective Teaching project, is taking a very thoughtful approach to teacher assessment by looking at multiple measures of student achievement and linking other metrics (e.g., classroom observations, teachers’ analyses of student work and their own teaching, and levels of student engagement) to capture a more robust and accurate view of who is effective and why.
If policymakers and the media mavens really want to learn about effective teacher evaluation, it’s time they turn to the wisdom of those who actually teach — like Larry Ferlazzo, a Teacher Leaders Network member, who penned a thoughtful response in this morning’s Washington Post blog The Answer Sheet to what will surely be a journalistic debacle.
I suspect the Los Angeles Times will soon hear from many teachers branded as “ineffective” who turn out to be “false negatives” (or a Type II error in statistical lingo), and the Times editors will need to re-think their approach to reporting on the debates about effective teaching.
I’m frankly startled by the Times's decision to publish this highly questionable data and risk humiliating many competent teachers by doing so. I wonder how journalists of national stature, who are trained to look at multiple sides of an issue and to be sure of their facts, could be so reckless?
Important post Barnett. I think what we have here is a classic example of using an instrument to determine something that it was not designed to assess. The results of the LA scores are assessment of what students expressed on a test, not teacher effectiveness. A better assessment is actually contained within the article.
John Smith says, ""Obviously what I need to do is to look at what I'm doing and take some steps to make sure something changes,".
A good teacher is one who makes that statement and then actually changes their practice. Smith might actually be a good teacher, if he can do what he says he must. When this is the measure then it becomes more complicated than a value-added math fraction, it becomes about differentiated professional development, adequate compensation for those like Aguilar who are high performing, and empowering a teacher to change, even if that means finding other roles within a school that a teacher can be successful at. Perhaps Smith has a hidden talent that he hasn't shared because he only does one thing all day. We may never know.
Posted by: J.M. Holland | August 18, 2010 at 11:43 AM
"Debacle in the making" is right. Now the mainstream press from outside of LA needs to follow up and document the debacle that I also predict will be made. The stakes have just grown higher sending the message that humilating teachers is the goal.
I predict that this controversary will not help identify effective and ineffective teachers very much. But it will identify "reformers" who really believe that VAMs will benefit teaching versus "reformers" who just want to destroy "the staus quo."
Posted by: john | August 18, 2010 at 06:41 PM
I wrote the following in response to John Merrow's Blog applauding the LA times - it gives a teacher's in the trenches view.
I need to take a deep breath and explain my concerns about the emphasis of standardized tests as a way to measure teacher effectiveness. Yes, undoubtedly there are differences in the quality of teachers - I’ve witnessed this as a teacher and as a parent of school age children. Yes, ineffective teachers should not be in teaching. As a matter of fact, individuals who do not exhibit intellectual and pedagogical ability should never be allowed into the profession. My concern, and it is a strong concern, is how we measure “effectiveness.” What makes a good teacher? This is an important question that I believe must be answered before we come up with a measurement for it.
I am a third grade teacher and I worry that my answer to this question does not jive with Arne Duncan’s and many others who make education policy decisions, and I worry about how this conflict will impact the profession I love so deeply. I became a teacher for the reasons Sam Chaltrain gives in his lovely blog – to help students “unlock the mystery of who they are by acquiring the skills and self-confidence they need to be seen and heard (at college, in their careers, and as citizens in a democracy) in meaningful, responsible ways.” Yes, in part this goal can be measured by my student’s growth on standardized tests, but SO SO SO much of what I do in my classroom, and what I do outside my classroom to support other students, parents and my colleagues, can not be reflected on one-shot quantitative tests.
I fear that if my role of a teacher is defined simply by standardized tests scores (such a narrow view of what I do) I will have to abandon my career before I relinquish my values. Please do not think that my students have performed badly on these tests over the years! Quite the contrary! My scores have been good, but I am unwilling to have my professional ability valued on such a meager scale. Review my scores but also review my portfolio, survey my students and parents, come and observe my lessons, my classroom environment, my commitment to my school and my drive to help other teachers excel! Etc. Etc. Unfortunately such a well-rounded review of my professional performance is neither as simple nor cheap as standardized testing.
What I do is so much more than prepare children for a test. However, if there is a chance that my picture would show up in the Boston Globe, I am afraid that I could be driven to take a more “Kaplanesque” approach to teaching. If I am being paid to increase tests scores – shouldn’t my focus be on test scores? I fear it would be – and then I would have to leave teaching.
Posted by: Liz Wisniewski | August 19, 2010 at 11:10 AM
Important contributions from all. I told one reporter the other day that the LA Times may very well be releasing value-added effects data on teachers who are teaching out of field. In doing so they will be indiscriminately labeling them ineffective - ignoring a important context variable. We are conducting research in CA and finding many teachers are forced to teacher subjects and grades for which they little or no experience. One teachers who was effective as a kindergarten teacher all of a sudden is placed in a 6th grade math class - and because of little prep and support does not produce much student achievement gains in her new assignment. And the LA Times will label this teacher as a failure - when it fact it is a system failure.
Posted by: Barnett Berry | August 20, 2010 at 06:01 AM
Evidence shows clearly what most people know intuitively: teachers matter more to student learning than anything else inside a school. Yet the design of our school systems fails to value and support effective teaching, improve overall teacher effectiveness, or recognize and reward those who take on and excel at the most difficult teaching assignments.
Posted by: Bachelor of Health and Movement (sport) | November 15, 2011 at 01:50 AM
A good teacher is one who makes that statement and then actually changes their practice. Smith might actually be a good teacher, if he can do what he says he must. When this is the measure then it becomes more complicated than a value-added math fraction, it becomes about differentiated professional development, adequate compensation for those like Aguilar who are high performing, and empowering a teacher to change, even if that means finding other roles within a school that a teacher can be successful at.
Posted by: smsf | November 15, 2011 at 02:42 AM