Testing the Testing

Gary Rubinstein has a great series of posts exploring the recently released evaluation data for New York City teachers.  His analysis has some bad news for the value-added testing metrics of the city: these metrics might not necessarily mirror real-world qualities, suggesting that analyzing teacher performance through these metrics might be a problematic enterprise at best.

Consider this finding: based on New York's assessment methods, there is hardly any correlation between a teacher's success in teaching a subject at one grade level and his or her success at teaching that same subject in the next grade level.  So a teacher who is supposedly successful teaching sixth-grade math could easily be one of the supposedly worst at teaching seventh-grade math.  Does that seem particularly plausible?  It might happen for some teachers, but Rubinstein finds almost no correlation for all teachers taken as a group.

Moreover, Rubinstein finds relatively little correlation between how successful a teacher is one year and how successful he or she is the next year.  With such variation, this testing regime might have a few methodological problems.