"Using Fidelity Data to Elucidate the Results of a Quasi-Experiment on Assessment-2-Instruction Technology"

SREE 2017 Spring Conference
March 1-4, Washington, DC

"Using Fidelity Data to Elucidate the Results of a Quasi-Experiment on Assessment-2-Instruction Technology"
Authors: Carol Connor & Karen Taylor


The current study represents the culmination of the 3-year development study funded by IES. During the first two years of the study, we re-designed and programmed A2i in collaboration with our partner schools. In year 3, we conducted a quasi-experiment to evaluate the efficacy of the newly designed ISI/A2i and to examine how teachers used A2i and how well they implemented ISI in the classroom. The research was conducted in two Title 1 schools in Arizona located in the rural fringe of Phoenix. The study included 898 students in K-3rd grade in 17 classrooms. We used a quasi-experimental delayed treatment design: one school was randomly assigned to use ISI/A2i at the beginning of the 2015-16 school year and the other school began using ISI/A2i in March of the same school year. Analysis of assessments in the fall, using multi-level models (HLM) revealed no significant differences in the immediate and delayed treatment students’ baseline performance on the Woodcock Johnson-III Letter-Word Identification test (M=371 and 367), the Passage Comprehension test (m=412 and 409) and the Gates MacGinitie Reading Test (M=425 and 424). Analysis of results using the March assessments (the end of the quasi-experiment) using HLM with the treatment variable (1=immediate; 0=delayed treatment) entered at the classroom level revealed significant effects of ISI/A2i in kindergarten and second grade but not in first or third grade. Analysis of A2i user log data revealed that kindergarten and second grade teachers used A2i for significantly more minutes from October to March (the length of the study) than did the first and third grade teachers. The results of this study demonstrate the utility of fidelity data to help in understanding the results of this quasi-experiment. The effects of ISI/A2i were greater when teachers actually used the technology.