Using Anchoring Vignettes to Calibrate Teachers Self-Assessment of TeachingReport as inadecuate

Using Anchoring Vignettes to Calibrate Teachers Self-Assessment of Teaching - Download this document for free, or read online. Document in PDF available to download.

Society for Research on Educational Effectiveness

High-quality measures of instructional practice are essential for research and evaluation of innovative instructional policies and programs, as well as for providing feedback to teachers and administrators. In this study, the authors examined whether using anchoring vignettes in web-based surveys improved the validity of teachers' self-assessments of their mathematics instruction. To investigate validity, they compared correlations between teachers' self-ratings and other measures of teaching including teachers' value-added scores, student surveys, and observation ratings of instruction before and after calibration to examine whether calibration improves the correlation between teachers' self-ratings and other teaching measures. Data came from 61 mathematics teachers in grades 4-9 participating in the Bill & Melinda Gates Foundation's Measures of Effective Teaching Extension project. The sample was roughly evenly distributed between elementary (grades 4-5) and middle school (grades 6-8) teachers, with only three percent teaching 9th grade. The authors used both non-parametric and parametric methods to calibrate teachers' self-ratings. Preliminary findings suggest that anchoring vignettes represent a promising innovation for measuring teachers' instruction through survey self-reports. Specifically, the authors found: (1) Teachers' survey responses that are calibrated through the use of anchoring vignettes have increased variation compared to teachers' raw survey responses, particularly for the cognitive challenge dimension; (2) Teachers' calibrated survey responses regarding mathematical vocabulary and cognitively challenging tasks are more strongly correlated with the composite measure of teacher performance compared to raw survey responses; and (3) If teachers gave their self-rating after rating the vignettes, rather than before, the entire collection of calibrated self-ratings are significantly correlated with the composite performance measure (p<0.05).

Descriptors: Self Evaluation (Individuals), Mathematics Teachers, Vignettes, Online Surveys, Teacher Surveys, Validity, Mathematics Instruction, Correlation, Comparative Analysis, Teacher Evaluation, Value Added Models, Student Surveys, Observation, Elementary School Teachers, Middle School Teachers, Secondary School Teachers, Grade 4, Grade 5, Grade 6, Grade 7, Grade 8, Grade 9, Statistical Analysis

Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries[at]; Web site:

Author: Yuan, Kun; Engberg, John; Kaufman, Julia; Hamilton, Laura; Hill, Heather; Umland, Kristin; McCaffrey, Daniel



Related documents