News
"Assessing the Correspondence between Student-Level Fidelity Data and Observational Measures of Classroom Instructional Content"

SREE 2017 Spring Conference
March 1-4, Washington, DC

"Assessing the Correspondence between Student-Level Fidelity Data and Observational Measures of Classroom Instructional Content"
Authors: Joshua Lawrence, Alex Lin, & Catherine Snow

Description

A previous analysis of notebooks collected from students participating in a randomized trial of the Word Generation program suggested that implementation was inconsistent across time and across content areas (WG is implemented in middle school classroom by math, science, social studies and English teachers). However, data collected from classroom observations seemed to suggest robust implementation. The current paper examines the data from these two sources to answer the following questions:

  1. How well was Word Generation implemented according to each measure?
  2. Which content areas implemented with greatest fidelity?
  3. How are the data from observations and notebook coding related?
  4. Which measures best explained variance in school pre-to-post performance?

Three large urban districts participated in this evaluation study. Only data contributed by teachers and students in the treatment schools were used in this analysis.

Our findings suggest that observations may be poor measures of fidelity, especially for programs, like Word Generation, that emphasize consistent weekly use. We suspect that classroom observers influenced practice, leaving notebook data as a better indicator. The notebook data tell us that most schools did not implement the program consistently whereas those that did saw impressive improvements. Completing 60% rather than 30% of the writing produces a 1.4-point improvement (.4 SD difference) in school level vocabulary posttest scores.