Evaluation of Short- and Long-term Effects of Reading Recovery in Montgomery County Public Schools

Transcription

1 Evlution of Short- nd Long-term Effects of Reding Recovery in Montgomery County ublic Schools June 2008 Nymbur Susn Min, h.d.

2 OFFICE OF SHARED ACCOUNTABILITY Dr. Stcy L. Scott, Associte Superintendent 850 Hungerford Drive Rockville, Mrylnd Dr. Jerry D. West Superintendent of Schools Dr. Fried K. Lcey Deputy Superintendent of Schools

3 Tble of Contents Executive Summry... vi Introduction... 1 The Reding Recovery Intervention... 1 Reding Recovery in Montgomery County ublic Schools... 1 Review of Literture... 3 Evlution Design... 4 Methodology... 5 Smples of Students... 5 Student Achievement Mesures... 6 Anlysis rocedures... 7 Dt Avilbility... 7 Results... 8 Evlution Question 1. Wht re the Short-nd Long-term Impcts of Reding Recovery on Students Who Successfully Completed Reding Recovery Compred with Rndom Smple of Grde-level eers?... 8 Short-term Impct... 8 Long-term Impct Evlution Question 2. How Do Students Who Successfully Complete Reding Recovery erform on Locl nd Stte Assessments Compred With Grde-level eers Who Were Similr but not Served in Reding Recovery? Short-term Impct Long-term Impct Discussion Conclusion Recommendtions References Appendix A: Elementry Schools Implementing Reding Recovery Appendix B: Demogrphic Chrcteristics of Students by Cohort Appendix C: Vribles in the Anlysis of Covrince Model rogrm Evlution Unit Reding Recovery Evlution i

4 Appendix D: Clcultion of Effect Sizes Appendix E: Reding Outcomes for Rndom nd Reding Recovery Students Appendix F: Grde 2 Outcomes for Rndom nd Reding Recovery Students Appendix G: Outcomes for Reding Recovery nd non-reding Recovery Students Appendix H: Assessment Dt rogrm Evlution Unit Reding Recovery Evlution ii

5 List of Tbles Tble 1 End-of-rogrm Sttus for Montgomery County ublic Schools Reding Recovery Students... 3 Tble 2 Assessment Dt Compiled by Cohort Yer nd Grde... 6 Tble 3 ercentge of Students Meeting Grde 1 End-of-Yer Reding Benchmrk by Subgroup... 9 Tble 4 roportion of Students Achieving Grde 2 End-of-Yer Benchmrk by Subgroup Tble 5 roportion of Students t or Above 50 th Ntionl Curve Equivlent for TerrNov Second Edition Assessments by Subgroup Tble 6 Mesures of Effect Sizes for Assocition Between Meeting Grde 1 Benchmrk nd rticiption in Reding Recovery by Cohort Tble 7 Adjusted Mens nd Men Differences for Grde 1 nd Mesures of Effect Sizes for Assocition Between Grde 1 Spring Text Reding Comprehension Levels nd rticiption in Reding Recovery by Cohort Tble 8 Mesures of Effect Sizes for Assocition Between Grde 2 End-of-yer Benchmrk Sttus nd rticiption in Reding Recovery by Cohort Tble 9 Adjusted Mens nd Men Differences for Grde 2 Text Reding Comprehension Levels nd Mesures of Effect Sizes for Assocition Between Reding Achievement nd rticiption in Reding Recovery by Cohort Tble 10 Mesures of Effect Sizes for Assocition Between Scoring t or Above the TerrNov Second Edition 50th Ntionl Curve Equivlent nd rticiption in Reding Recovery by Cohort Tble 11 Adjusted Mens nd Men Differences for TerrNov Second Edition Mthemtics Scle Scores nd Mesures of Effect Sizes for Assocition Between Reding Achievement nd rticiption in Reding Recovery Tble 12 Mesures of Effect Sizes for Assocition Between roficiency on the Mrylnd School Assessment nd rticiption in Reding Recovery by Cohort rogrm Evlution Unit Reding Recovery Evlution iii

6 Tble 13 Adjusted Mens nd Men Differences for Grde 3 Reding Mesures nd Mesures of Effect Sizes for Assocition Between Reding Achievement nd rticiption in Reding Recovery by Cohort rogrm Evlution Unit Reding Recovery Evlution iv

7 List of Figures Figure 1 roportion of students t or bove Grde 1 end-of-yer benchmrk by cohort yer... 8 Figure 2 roportion of students chieving Grde 2 end-of-yer benchmrk Figure 3 roportion of rndom smple nd reding recovery students t or bove the TerrNov Second Edition 50 th Ntionl Curve Equivlent Figure 4 roportion of students performing t or bove proficiency on Grde 3 reding Mrylnd School Assessment Figure 5 roportion of students performing t or bove proficiency on reding Mrylnd School Assessment by rce/ethnicity Figure 6 roportion of students performing t or bove proficiency on the Grde 3 reding Mrylnd School Assessment by service receipt Figure 7 roportion of Reding Recovery nd non-reding Recovery students meeting the Grde 1 end-of-yer benchmrk Figure 8 roportion of Reding Recovery nd non-reding Recovery students meeting the Grde 2 end-of-yer benchmrk Figure 9 roportion of Reding Recovery nd non-reding Recovery students students t or bove the TerrNov Second Edition 50 th Ntionl Curve Equivlent Figure 10 ercentge proficient nd bove on the Grde 3 reding Mrylnd School Assessment rogrm Evlution Unit Reding Recovery Evlution v

8 Executive Summry This report presents findings on the short- nd long-term impct of the Reding Recovery (RR) intervention on student litercy chievement. RR, reserch-bsed erly intervention progrm is one of the trgeted inititives implemented in Montgomery County ublic Schools (MCS) since 1997 to improve student chievement in elementry schools. This inititive supports Gol 1 of the MCS strtegic pln Our Cll to Action: ursuit of Excellence (MCS, 2007). Delivered in Grde 1, the RR intervention is for those students reding below grde level, lwys mong the lowest 25% of the clss. The intervention is designed to ccelerte the litercy skills of the lowest performing Grde 1 students from the bottom of their clss to the verge, where they cn profit from regulr clssroom instruction. Approximtely students prticipte in RR in given yer. The following two questions guided the evlution study: 1. Wht re the short- nd long-term impcts of RR on students who successfully complete the progrm, s mesured by locl nd stte ssessments compred with rndom smple of grde-level peers? 2. How do students who successfully complete RR perform on locl nd stte ssessments compred with grde-level peers who were reding below grde level by the end of kindergrten or fll of Grde 1 (similr in initil reding bility to RR students) but not served in RR? Achievement dt for three successive cohorts of RR students (cohort 1), (cohort 2), nd (cohort 3) were used to ddress the evlution questions. The performnce of RR prticipnts ws compred with two groups: strtified rndomly smpled grde-level peers nd students similr in reding bility to RR prticipnts but who did not prticipte in RR. To ssess the short-term impct, the study included nlyses of the verge performnce of RR nd comprison students within ech cohort t the end of Grde 1. To ssess long-term impct, the performnce of RR students ws compred with the performnce of students in the comprison groups t the end of Grdes 2 nd 3. Key Findings Short-term impct The short-term impct of prticipting in RR ws positive. Within cohorts, the performnce of RR students on spring Grde 1 text reding comprehension levels nd rtes of meeting the MCS Grde 1 end-of-yer benchmrk ws comprble with or significntly exceeded the performnce of their rndom smple grde-level peers. The extent of the effect of ssocition between prticiption in RR nd Grde 1 reding chievement vried depending on cohorts nd mesures exmined nd the effect ws strongest for students in cohorts 2 nd 3. Also, medium to lrge effect sizes for students in cohorts 1, 2, nd 3 confirmed the positive effect of RR. rogrm Evlution Unit Reding Recovery Evlution vi

9 Within ech cohort, RR students s group showed less vrition in their performnce on the Grde 1 reding mesures thn students in the rndom smple comprison group. The spring text reding comprehension levels nd the rtes of meeting the Grde 1 endof-yer benchmrk for RR students were significntly higher thn levels for students reding below grde level by spring of kindergrten nd fll of Grde 1 (similr peers) who did not receive RR. The corresponding lrge nd medium effect sizes for ssocition between prticiption in RR nd reding chievement for students in cohorts 1, 2, nd 3 confirmed the mrked effect of RR. Long-term impct Contrry to number of longitudinl studies conducted in the United Sttes, the study did not find sustined effects of RR in Grde 2. The findings for Grde 2 reding outcomes vry by mesure nd cohort being exmined. Overll, cross cohorts, only one third of RR students met the Grde 2 end-of-yer reding benchmrk, nd even fewer students plced t or bove the 50 th Ntionl Curve Equivlent (NCE) for TerrNov Second Edition (TN/2) ssessments. Contrry to the expecttion tht RR students would perform t comprble levels with their grde-level peers, cross the three cohorts sttisticlly significnt differences for Grde 2 spring text-reding levels nd TN/2 scle scores were found in fvor of rndomly smpled peers. The smll negtive effect sizes for Grde 2 reding mesures confirmed tht the performnce of the RR students lgged behind tht of their rndom smple grde-level peers. At the sme time, the performnce of RR students on the Grde 2 text reding comprehension levels ws comprble with or lower thn the performnce of similr students who did not receive RR. The negligible effect sizes signify tht ny slight differences were not meningful in n eductionl setting. Furthermore, significnt differences in the TN/2 scores between RR nd similr students were found in fvor of similr students in ll three cohorts. The relted smll to medium negtive effect sizes for ssocition between prticiption in RR nd TN/2 scores verify tht the performnce of students in RR lgged behind the performnce of similr students not in RR. The findings for Grde 3 reding outcomes vry by mesure nd cohort being exmined. In generl, more one thn hlf of the RR students were t or bove proficiency on the Grde 3 Mrylnd Stte Assessment (MSA) in reding. The Grde 3 reding MSA proficiency rtes for RR students in cohort 1 were significntly lower thn the rtes for their rndomly smpled peers. Also, the difference in djusted MSA reding scle scores for RR nd rndom smple peers in cohorts 1 nd 2 were sttisticlly significnt in fvor of rndomly smpled students. The smll negtive effect sizes confirm tht the performnce of RR students lgged behind tht of their rndom smple grde-level peers. Moreover, the difference in djusted MSA scle scores for RR nd similr students in cohort 1 ws not sttisticlly significnt. Negligible effect sizes confirm tht the performnce of cohort 1 RR students ws comprble with similr students who did not prticipte in RR. On the other hnd, the differences in djusted men MSA scores for RR nd similr students in cohort 2 were significntly different in fvor of similr students who did not prticipte in RR. The rogrm Evlution Unit Reding Recovery Evlution vii

10 smll negtive effect sizes confirm tht the performnce of RR students on the Grde 3 reding MSA lgged behind tht of similr grde-level peers who did not receive RR. Significnt differences in Grde 3 spring Mesures of rogress in Reding (RIT) scores were found in fvor of rndom smple students in cohorts 1 nd 2. The medium effect sizes confirm tht the performnce of RR students lgged behind tht of their rndom smple peers. Moreover, significnt differences in Grde 3 spring RIT scores were found in fvor of similr students in cohort 2 but not for cohort 1. The smll negtive effect sizes confirm tht the performnce of cohort 2 RR students lgged behind tht of similr grde-level peers who did not receive RR. Recommendtions Investigte fctors tht contribute to low reding proficiency levels of RR students nd students receiving specil services on the Grde 2 end-of-yer benchmrk nd TN/2 ssessments. Resons for RR students flling behind in Grde 2 re uncler, prticulrly since the Grde 2 findings counter findings reported by others (United Sttes Deprtment of Eduction (USDE, 2007). In generl, districtwide trends show tht less thn 50% of students receiving specil services nd students from the Africn Americn nd Hispnic subgroups meet the Grde 2 end-of-yer reding benchmrk or score bove the 50 th NCE on the TN/2. Furthermore, the gretest vribility mong schools nd within subgroups in the district occurs for the Grde 2 end-of-yer reding benchmrk nd on TN/2 (Chesney & Fink, 2007; Dougls, 2006; Zho & Von Secker, 2007). Therefore, understnding nd ddressing issues relted to reding performnce in Grde 2 is criticl for closing the chievement gp mong student subgroups nd schools. Estblish trgeted trining of Grde 2 techers nd reding specilists to support former RR students nd students receiving specil services. Consider the development of pproprite Grde 2 reding intervention progrms for former RR students nd other low-performing students. In prticulr, pproprite interventions mtched with specific needs of students for schools with less thn 75% of students chieving the spring Grde 1 nd 2 end-of-yer reding benchmrk re needed. Add mrker to Online Administrtive Students Informtion System (OASIS) to identify students who received RR ech yer. This will enble schools nd reserchers to trck nd monitor performnce of RR students over time. Monitor progress of RR students in Grde 2 nd provide dditionl support s needed. Eliciting clssroom techers observtions of RR students before RR, chnges in students fter RR, nd res of further needs for RR students fter the intervention will help techers trget their support to the specific needs of the students t ech stge. rogrm Evlution Unit Reding Recovery Evlution viii

11 Evlution of Short- nd Long-term Effects of Reding Recovery in Montgomery County ublic Schools Nymbur Susn Min, h.d i. Introduction Reding Recovery (RR) is one of the trgeted inititives implemented in Montgomery County ublic Schools (MCS) since 1997 to improve student chievement in elementry schools. These inititives support Gol 1, Ensure success for ll students, of the MCS strtegic pln Our Cll to Action: ursuit of Excellence (MCS, 2007). Delivered in Grde 1, the gol of RR is to provide intensive support to the lowest chieving first grders to enble them to ttin grdelevel litercy skills. During the school yer, RR ws implemented in 33 elementry schools. Between students receive RR in given yer. The Reding Recovery Intervention The RR intervention is designed to ccelerte the reding nd writing performnce of the lowest performing Grde 1 students in short time from the bottom of their clss to the verge, where they cn profit from regulr clssroom instruction. The first expected outcome of RR is tht most students ccelerte their lerning so tht they re ble to lern in the clssroom nd meet grde-level expecttions. The second expected outcome of RR is tht the children who need long-term lerning support re identified nd referred for further evlution. rior to the intervention, n Observtion Survey (OS) of Erly Litercy Achievement is dministered to the lowest chieving first grders. The OS includes six tsks: Letter Identifiction, Ohio Word Test, Concepts bout rint, Writing Vocbulry, Hering nd Recording Sounds in Words, nd Text Reding (Cly, 2002). During the intervention, individul students receive hlf-hour lesson with specilly trined RR techers dily for weeks. As soon s the students ccelerte their reding to be within the verge rnge in their clss nd demonstrte tht they cn continue to chieve t or bove grde level, their dily RR lessons re discontinued nd new students begin individul instruction. The six tsks of the OS lso re dministered to RR students t the conclusion of the progrm. Yer-end OS scores serve s the posttest mesures in judging the progress mde by RR students. In ddition to strong performnce on the OS, the students who successfully complete RR re expected to demonstrte the bility to work well within their clssroom settings. Reding Recovery in Montgomery County ublic Schools Setting. Since 1999, ll federlly funded Title I schools (with the exception of Reding First schools 1 ) with Grde 1 students re stffed with t lest one.5 full-time equivlent (FTE) RR techer. In ddition, non-title I schools re llocted.5 RR techer bsed on need, s identified by district dt. By the school yer, 17 Title I schools nd 16 non-title I schools implemented RR (See Appendix A). Also, some schools elect to use.5 supplementl position (such s focus techer or cdemic intervention techer) for RR. 1 Highlnd, Rosemont, Summit Hll, nd Wheton Woods elementry schools rogrm Evlution Unit Reding Recovery Evlution 1

12 In ddition, MCS is n uthorized site for RR through Ohio Stte University. The MCS RR techer trining site is housed t Brod Acres Elementry School. One full-time RR techer leder provides the trining for techers nd coordintes the MCS RR progrm. Eligibility for RR. Kindergrten techers complete rnking sheets in June, listing their current students litercy skills from high to low, bsed on the MCS Assessment rogrm Reding (mclss 3D) nd informl clssroom litercy ssessments. RR techers first use the kindergrten rnking sheets to select the lowest children to begin testing for RR on the morning of the first dy of school. Grde 1 techers re sked to recommend ny other students they re concerned bout nd ny students new to the school. No child is excluded from screening, except those who hve been retined in Grde 1 or those with Individul Eductionl rogrms (IEs) for reding nd writing. Accordingly, RR techers mke sure they re dministering the OS to pproximtely 25% 30% of the Grde 1 students they serve. The dt collected re recorded on selection sheets for the selection meeting for first-round students. There is no preestblished cut-off score. The lowest students re selected first, looking cross ll six ssessments of the OS (Rndll, 2007). The MCS RR site techer leder collects nd evlutes ll selection sheets from ech school to verify tht the students selected re indeed the lowest performing students, bsed on ssessment dt in ech school. The RR school tem mkes preliminry decision regrding who is served. Level of coverge. Level of coverge refers to the percentge of Grde 1 students served by RR. Full implementtion of the progrm requires enough RR teching time to ensure tht ll Grde 1 students, in need of dditionl ssistnce (pproximtely 25%), hve ccess to RR (Rndll, 2007). Full implementtion lso mens every child needing RR hs the opportunity to be served. 2 The mjority of the schools were prtilly implementing the intervention. End-of-progrm sttus. At the end of the intervention, RR students re ssigned to one of the following end-of-progrm sttus ctegories: Discontinued: A child who successfully met the criteri to be discontinued from the intervention during the school yer or t the time of yer-end testing. Recommended Action After Complete Intervention: A child who ws recommended by RR professionls for ssessment/considertion of other instructionl support t the point of deprture from RR, fter receiving full series of lessons of t lest 20 weeks. Incomplete rogrm t Yer-End: A child who ws still in RR t the end of the school yer with insufficient time (less thn 20 weeks) to complete the intervention. Moved While Being Served: A child who moved out of the school while being served before specific progrm sttus could be determined nd who my or my not hve hd full progrm of 20 weeks. None of the Above: A rre ctegory used only for child who ws removed from RR under unusul circumstnces, with fewer thn 20 weeks of instruction (Rndll, 2004). Tble 1 presents informtion regrding end-of-progrm sttus for MCS students who received RR during the through school yers. More thn one hlf of students were : Brod Acres nd New Hmpshire Esttes; : Brod Acres, New Hmpshire Esttes, nd Rolling Terrce; : New Hmpshire Esttes nd Rolling Terrce. rogrm Evlution Unit Reding Recovery Evlution 2

13 successfully discontinued from the progrm ech yer, rnging from 55% 66%. The ntionl verge is 58% 59% for the sme period (Rndll, 2005; 2006). Tble 1 End-of-rogrm Sttus for Montgomery County ublic Schools Reding Recovery Students Cohort Sttus n % n % n % Discontinued Recommended ction fter full progrm of 20 weeks or more Incomplete progrm t yer end Moved while being served None of the bove Totl Source: RR progrm dt. Columns do not dd up to due to rounding. Review of Literture Evidence from literture points out tht RR provides compenstory reding benefits, reduces the chievement gps tht exist long rcil nd socioeconomic lines, nd provides ccelerted reding benefits (Schmitt & Gregory, 2001; West, Denton & Reney, 2000; Swrtz & Hoffmn, 1993; innell, 1989; D Agostino & Murphy, 2004). Compenstory reding benefit. Studies by Schmitt nd Gregory (2001; 2005) demonstrted tht the mjority of rndomly selected RR children were performing s well s rndom smple of their grde-level peers on the vocbulry nd comprehension subtests of the Gtes-McGinitie Reding Test nd running records of orl text reding on the OS. Similrly, Grde 3 students results on the Comprehensive Test of Bsic Skills-5/TerrNov Form B reflected norml distribution with men t the 45 th percentile for those students who were the lowest chieving first grders before receiving the progrm. Reduction in chievement gps long rcil nd socioeconomic lines. Mounting sttisticl evidence continues to document tht chievement gps lie between the rces nd lso between more or less economiclly disdvntged students (Lee, 2002; West, Denton & Reney, 2000). On the bsis of wht hs been observed bout the chievement gp nd who is ffected, the expecttion is tht not only will children fll behind their peers long rcil nd economic lines but lso this gp my sty with them over time. One study evluted whether students who receive RR mke differentil progress long rcil nd economic lines (Rodgers, Wng, & Gomez-Bellenge, 2004). The findings concur with those of innell et l. (1994) nd indicte tht n erly intervention such s RR is sufficient to mke difference in closing the litercy gp tht exists long rcil nd economic lines. Becuse of n erly intervention such s RR, these findings suggest tht rce nd socioeconomic fctors cesed to be significnt predictors of the reding success of Grde 1 students. In ddition, findings by Ashdown & Simic (2000) suggest tht RR produces similr outcomes for students with different levels of English proficiency nd offers n pproprite solution for Grde 1 students experiencing problems in reding nd writing. rogrm Evlution Unit Reding Recovery Evlution 3

14 Accelerted reding benefit of RR. The success of RR in ccelerting reding hs been crefully documented (Swrtz & Hoffmn, 1993; Swrtz, 2005; innell, 1989; USDE, 2007). In these studies, students entered s nonreders, discontinued t level considered to be the end of Grde 1, nd, by the end of Grde 1, hd ccelerted their reding to level equivlent to Grde 2. Other studies indicte tht students who prticipted in RR mintined progress in Grdes 3, 4, nd 5 (Tlwr & Hill, 1993). innell (1989) lso reported tht Grde 4 RR students demonstrted tht they could ccurtely red text t or bove the Grde 6 level. Skepticism bout RR. Becuse it is widely studied nd well known, RR hs drwn criticism from severl reserchers nd eductors. Some critics ssert tht ) the RR dt reporting system is flwed; b) the stndrd for successful completion of RR is not equitble cross schools nd prticipnts; c) RR does not rise overll school chievement levels; d) some studies do not use strong cusl design, nd some reserchers report misleding success rtes; or e) some of the findings re derived from unorthodox reserch procedures (Elbum, Vugh, Huges, & Moddy, 2000; Shnhn & Brr, 1995; innell & Lyons, 1994). Overll, the evidence tht RR is effective in vriety of contexts fr outweighs the criticisms. A recent study reported tht RR hs outcomes of eductionl significnce for the mjority of the prticipting students (Timperley, Fung, Wilson & Brrr, 2006). Evlution Design This study follows qusi-experimentl design nd compres the chievement of students who did not prticipte in RR intervention with students who did. Specific evlution questions were s follows: 1. Wht re the short- nd long-term impcts of Reding Recovery on students who successfully completed Reding Recovery s mesured by locl nd stte ssessments compred with rndom smple of grde-level peers? Are there differentil ptterns of reding chievement mong student subgroups? 2. How do students who successfully complete RR perform on locl nd stte ssessments compred with grde-level peers who were similr in reding bility/eligible but were not served by RR intervention? For evlution question 1, if RR is effective, the expecttion is tht, fter the intervention, the performnce of RR students will be comprble with tht of their rndom smple peers. For evlution question 2, the expecttion is tht the performnce of the RR students will be better thn the performnce of similr students who did not prticipte in RR. Short-term impct denotes the performnce of students immeditely fter the intervention t the end of Grde 1. Long-term impct corresponds to the performnce of the students t the end of Grdes 2 nd 3, which re 2 nd 3 yers fter the RR intervention. rogrm Evlution Unit Reding Recovery Evlution 4

15 Methodology Smples of Students As noted erlier, students re not rndomly ssigned to RR. Consequently, experimentl designs re not possible to employ. A prtil solution is to use qusi-experimentl design, which includes n pproprite comprison group. This study lso used sttisticl controls to blnce the groups, due to lck of rndom ssignment of students to the tretment or comprison groups. To exmine whether ) RR students ctch up with their grde-level peers nd b) whether RR students perform better thn similr students who did not receive RR necessitted compring the performnce of RR students with two groups of students. The pproprite comprison groups re rndom smple of peers nd smple of students who were reding below grde-level by the strt of Grde 1 who did not prticipte in RR. Tretment groups (RR). The tretment groups in this study were three successive cohorts of students who completed nd were successfully discontinued from Reding Recovery during the , , nd school yers. Comprison groups. The following procedures were used to crete pproprite comprison groups. First, enrollment dt for ll students who were in Grde 1 during the , , nd school yers nd student demogrphic informtion pertining to rce/ethnicity, gender, specil services received (English for Spekers of Other Lnguges ESOL, Free nd Reduced-price Mels Services FARMS, nd specil eduction services) nd verge ttendnce for given yer were compiled from dt source for yerly summry dt for students enrolled in MCS. Second, smpling technique ws used to drw the following strtified (by ESOL, FARMS, rce/ethnicity, nd gender) rndom smples of grde-level peers: 1. Rndom groups. This included rndomly selected students who were not exposed to RR t ny time. A smple ws drwn for ech cohort yer. These strtified rndom smples were used s the comprison groups for evlution question Non-RR groups. To form n pproprite comprison group for ddressing question 2, rndomly selected smple of students ws drwn from group of students of similr demogrphic profile nd reding bility to those enrolled in RR. These students were drwn from dt set of students who were reding below grde level by spring of kindergrten nd fll of Grde 1 t risk of reding filure. These students re defined s those who were reding t Text Reding Comprehension (TRC) level of 0 2 t the end of kindergrten. These students did not receive RR becuse ) there ws no RR intervention progrm in their schools or b) if their school implemented RR, the implementtion ws prtil nd these students were not rnked mong the lowest 25% in their clss. A smple ws drwn for ech cohort yer. These smples were used for nlyses relted to evlution question 2. The strtified rndom nd non-rr comprison groups closely mtched nd corresponded to the number nd profile of students enrolled in RR in given yer on the bsis of rce/ethnicity, gender, FARMS, nd ESOL sttus (See Appendix B). rogrm Evlution Unit Reding Recovery Evlution 5

16 Student Achievement Mesures Locl ssessments. The following locl ssessments were used in this evlution: 1. Montgomery County ublic Schools Assessment rogrm-rimry Reding (MCSA-R). MCSA-R is designed to mesure foundtionl reding skills nd reding proficiency in kindergrten, Grde 1, nd Grde 2. For this evlution, the outcome mesures of Grdes 1 nd 2 end-of-yer benchmrk sttus nd the students TRC levels were used. The MCS end-of-yer benchmrks re specified s follows:. Grde 1 reding t TRC level 16 or higher with 90% ccurcy nd chieving score of 4 out of 5 on orl comprehension. b. Grde 2 reding t TRC level M (27) or higher with 90% ccurcy, hving n orl comprehension score of 4 out of 5, nd scoring t lest 2 out of 3 on two written comprehension items (Chesney & Fink, 2007). 2. TN/2. 3 Grde 2 NCE nd reding scle scores were nlyzed. The ntionl norm or verge score for ll TN/2 tests is the 50 th NCE (Zho &Von Secker, 2007). 3. Mesures of Acdemic rogress Assessment in Reding (MA-R). Grde nd 2007 MA-R Rsch Unit (RIT) scores were used for the nlyses mesurement scle developed to simplify the interprettion of test scores. RIT scores rnge from bout 150 to 300. RIT scores mke it possible to follow student s eductionl growth from yer to yer. Stte ssessments. Grde 3 test scores on the 2006 nd 2007 Mrylnd School Assessment (MSA) reding subtest nd the corresponding reding proficiency levels were used in the evlution. Cohort 3 ws in Grde 2 in nd their MSA scores were not vilble becuse this is not given until spring semester Grde 3. Tble 2 presents the dt on the yer ech cohort prticipted in RR instruction nd the informtion compiled for ech cohort of students. Tble 2 Assessment Dt Compiled by Cohort Yer nd Grde Reding Recovery rticiption Yer Grde s of Avilble stte nd district ssessment dt nd MCSA-R, end-of-yer benchmrk (cohort 1) sttus, TRC level 2005 CTBS NCE nd scle scores 2006 MSA Grde 3 scle score nd proficiency level (cohort 2) (cohort 3) 2006 MA-R Grde nd MCSA-R, end-of-yer benchmrk sttus, TRC level 2006 TN/2 NCE nd reding scle score 2007 MSA Grde 3 scle score nd proficiency level 2007 MA-R Grde 3 scle score nd mclss 3D- MCSA-R, end-of-yer benchmrk sttus nd TRC level 2007 TN/2 reding NCE nd reding scle score 3 TN/2 is not loclly developed but is loclly dministered norm-referenced test. rogrm Evlution Unit Reding Recovery Evlution 6

17 Anlysis rocedures 4 The nlyses conducted for evlution questions 1 nd 2 re designed to ) describe the performnce of RR students reltive to the comprison group nd b) mesure the net short- nd long-term effects of the RR progrm on student chievement. Three different nlyticl procedures were pplied to ddress the questions: descriptive nlyses of performnce levels, nlyses of covrince (ANCOVA) for continuous outcomes, nd logistic regression models for ctegoricl outcomes mesures. The following prgrphs describe the nlyses used to ddress evlution questions 1 nd To sttisticlly control for the nonequivlence of the RR nd comprison groups nd to isolte the progrm effects of RR, propensity score ws computed. The propensity score is described s conditionl probbility tht student will be treted (be plced in RR) bsed on n observed group of covrites (Lueleen, Shdish & Clrk, 2006). ropensity scores creted by ESOL, FARMS, specil eduction sttus, nd rce/ethnicity, nd mesure of initil reding bility were estimted using logistic regression procedures. Becuse two groups could be similr on demogrphic vribles but dissimilr on cdemic mesures, mesure of initil reding bility by fll of Grde 1 ws included in the computtion. A seprte propensity score ws estimted for ech yerly cohort. Subsequently, the propensity scores were ctegorized into quintiles nd included in the ANCOVA nd logistic regression models. 2. ANCOVA nd logistic regression nlytic models were developed to isolte the effects of RR, while simultneously controlling for other preexisting or intervening vribles, such s prticiption in FARMS, ESOL, specil eduction services, rce/ethnicity, initil bility t ech grde level, plus propensity score. For ech reding chievement mesure, student dt were nlyzed seprtely for ech cohort nd t Grdes 1, 2, nd 3. The dependent vribles were the spring reding mesures t ech grde level. The independent vrible of RR sttus included two levels: prticiption or nonprticiption in RR (See Appendix C, Tbles C1 C2). 3. Effect sizes were computed to ssess the ssocition between prticiption in RR nd student chievement in reding or the prcticl significnce of the RR intervention for ech outcome mesure nd t ech grde level (See Appendix D). In evluting eductionl progrms, effect sizes of 0.20 nd bove re considered importnt by eductionl reserchers, i.e., the lrger the effect sizes, the stronger the impct of the intervention (Slvin, 1990; Cohen, 1988). Dt Avilbility Only students who were successfully discontinued from RR (Tble 1) were included in the nlyses. Also, to compre the proportion of students chieving certin grde-level benchmrk nd/or the performnce of students on other chievement mesures, the scores of students who hd vlid scores for specific mesure were included. Therefore, the number of students vried t ech step of the nlyses nd the sme students could not be compred t ech ssessment. 4 The uthor would like to thnk Dr. Scot McNry, ssocite professor, Towson University, for his ssistnce with reviewing the nlyticl procedures nd nlyses used in this study. rogrm Evlution Unit Reding Recovery Evlution 7

18 Results 5 The results re orgnized by evlution questions. Evlution Question 1. Wht re the short- nd long-term impcts of reding recovery on students who successfully completed reding recovery compred with rndom smple of grde-level peers? Are there differentil ptterns of reding chievement mong student subgroups? Short-term Impct The chievement dt on Grde 1 reding mesures were nlyzed in three wys. First, chievement ws considered reltive to grde-level MCSA-R benchmrk. More specificlly, these nlyses investigted wht percentge of students from the rndom nd RR smples met the spring TRC nd comprehension criteri for Grde 1 end-of-yer reding benchmrk. Second, summry sttistics of spring TRC levels were clculted. Third, while controlling for preexisting differences, the impct of RR on student performnce ws estimted. Grde 1 end-of-yer reding benchmrk. After prticiption in RR, students re expected to be reding t or bove grde level nd lso meet the reding benchmrk t the sme rte s their grde-level peers. Informtion on performnce of students on the Grde 1 end-of-yer benchmrk is presented in Figure % Meeting Benc hmrk Rndom RR Cohort Figure 1. roportion of students t or bove Grde 1 end-of-yer benchmrk by cohort yer. For students in cohorts 1, 2, nd 3, the pssing rtes for RR students on the Grde 1 end-of-yer benchmrk were significntly higher thn rtes for their rndom smple peers (p<0.05). 5 This study ssumes tht the RR intervention ws implemented s designed. rogrm Evlution Unit Reding Recovery Evlution 8

19 After controlling for preexisting differences, smll effect size ws found for cohort 1 students (0.27) whereby lrge nd moderte effect sizes for cohort 2 (0.87) nd cohort 3 (0.51) were found (Appendix E, Tble E1 through E1d). The smll to moderte to lrge effect sizes for cohorts 1, 2, nd 3 indicte tht the differences in the proportions for the two groups were eductionlly meningful. Tht is, the verge RR students in cohorts 1, 2, nd 3 performed t the 62 nd, 79 th, nd 69 th percentile of the rndom smple students, respectively. Grde 1 end-of-yer benchmrk by subgroup. Compring the performnce of students mong groups within cohorts, RR students in cohorts 2 nd 3 outperfo rmed their rndomly smpled grde-level peers nd this pttern of performnc e ws reflected for the rcil/ eth nic subgrou ps, except for Asin Americns. The lowest pssing r te for RR stud ents ws mong the Hispnic students in cohort 1 (Tble 3). Historicl trends indicte tht pssing rtes of students receiving s pecil servic es (e.g., ESOL, FARMS, nd specil educ tion) on the Grde 1 end-of-yer benchmr k re lowe r thn pssing rtes for students not receiv ing these services (Chesne y & Fink, 2007). The sme tre nds were observed for students in this study, with few exceptions: Cohort 2 RR students wh o received FARMS services outperformed their Non-FARMS counterprts. Additionlly, cohort 3 RR FARMS students lso outperfo rmed the rndom smpl e Non-FARMS peers. No tbly, for every cohort, RR students who received ESOL services outperformed their ESOL counterprts i n the rndomly selected smple (Tble 3). Tble 3 ercentge of Students Meet ing Gr de 1 End-of-Yer Re ding Bench mrk by Subgroup Cohort 1 ( ) Cohort 2 ( ) Cohort 3 ( ) Rndom RR Rndom RR Rndom RR Subgroups n % n % n % n % n % n % Rce/Ethnicity Africn Americn Asin Americn Hispnic White Gender Femle Mle Specil Services Non-FARMS FARMS Not ESOL ESOL Not Specil Ed Specil Ed Note. RR= Reding Recovery. Results for Americn Indin nd subgroups with fewer thn five students re not reported. Grde 1 spring TRC levels. The students in the RR smple strted Grde 1 with fll TRC levels tht were significntly lower thn those of their grde-level peers. While compring the performnce of students mong groups within ech cohort, the dt show tht men Grde 1 TRC levels for the students in RR nd rndom smples incresed substntilly nd becme very similr by spring of Grde 1 (See Appendix E, Tble E2). As grphiclly illustrted in Appendix rogrm Evlution Unit Reding Recovery Evlution 9

20 E, Figures E1 1c, pge 37, the distribution of RR students men spring Grde 1 TRC levels pproximted the spred of the spring TRC levels for students in the rndom smple. Clerly, the TRC levels of the RR students were no longer clustered t the low end of the distribution. Grde 1 spring TRC levels by subgroup. Across cohorts, the students in the RR smple mde verge gins of TRC levels compred with 9 10 TRC levels for their rndom smple counterprts. Ech rcil/ethnic group benefited from RR, with White nd Asin Americn students mking higher gins in men spring Grde 1 TRC levels thn other subgroups (See Appendix E, Tbles E3 E5). Asin Americn students in the cohort 1 RR smple did slightly better thn Asin Americns in the rndom smple, wheres those Asin Americn students in cohort 2 did not. The difference in performnce of Africn Americn students who got RR nd those who did not ws less thn one TRC level. White students who hd RR performed s well (cohort 1 nd 2) or better (cohort 3) thn White students in the rndom smple. The performnce of Hispnic students vried depending on the cohort. Additionlly, the performnce of cohort 1 RR Hispnic students ws comprble with the performnce of Hispnic students in the rndom smple; cohort 2 Hispnic students in the rndom smple did better thn Hispnic students in the RR smple; cohort 3 Hispnic students in the RR smple outperformed their rndom smple Hispnic counterprts. Grde 1 students re expected to be reding t TRC level 16 or bove by spring of Grde 1. To be performing s well or better thn the students in the rndom smple, the ANCOVA should show no significnt differences between the spring Grde 1 TRC levels for RR nd rndom students nd/or show significnt differences in fvor of the RR students. Also, the effect sizes for ssocition between prticiption in RR nd performnce on the reding mesures to ssess the prcticl significnce of the intervention would be expected to be round or higher thn zero. While controlling for preexisting differences, the results of the ANCOVA indicted significnt RR effect. As expected, the men spring TRC levels for students in the RR smples were comprble with or higher thn spring TRC levels for students in the rndom smple cross the three cohorts (See Appendix E, Tble E6). The results show smll to high effect sizes for ssocition between prticiption in RR nd the spring TRC levels for cohorts 1 (0.32), 2 (0.76), nd 3 (0.68). Tht is, the verge student in the RR smple would hve higher score thn 63% of cohort 1 students, 79% of cohort 2 students, nd 73% of cohort 3 students in the rndom smple (Springer, 2006; Becker, 2007). Short-term impct summry Across cohorts, the frequency distribution of spring Grde 1 TRC levels for students in RR nd the rndom smple ws comprble, indicting tht RR students were no longer clustered t the low end of the spring Grde 1 TRC distribution. In ddition, the mjority of students in the RR smple met or exceeded the Grde 1 end-of-yer benchmrk t rtes tht were comprble with or significntly higher thn rtes for the rndom smple students. As expected cross the three cohorts: the mens for the RR students were either comprble or exceeded the TRC levels for their rndomly smpled peers. The size of the effect of the intervention vried by cohort but ws in fvor of RR students in ll three cohorts. RR ws effective in rising the Grde 1 spring TRC levels nd pssing rtes on the Grde 1 end-of-yer benchmrk of struggling Grde 1 students to the level of their grde-level peers or better. rogrm Evlution Unit Reding Recovery Evlution 10

21 In ddition, RR students s group showed less vrition in their performnce on the spring Grde 1 TRC levels nd rtes of meeting the Grde 1 end-of-yer benchmrk thn their rndomly smpled peers. Also, for every cohort, RR students who lso received ESOL services outperformed their ESOL counterprts in the rndomly selected smple. Long-term Impct Long-term impct of the RR progrm for ech cohort ws investigted reltive to reding performnce in Grdes 2 nd 3. The strtified rndom smple of grde-level peers ws used s the comprison group in the nlyses of long-term impct for evlution question 1. Grde 2 end-of-yer reding benchmrk. The expecttion is tht RR students will continue to perform s well s their grde-level peers s they move on to higher grdes. While compring groups within ech cohort, the trends observed in Grde 1 were reversed for the Grde 2 MCSA-R spring benchmrk. The performnce of Grde 2 students vried by cohort nd ws mostly in fvor of rndomly smpled students. As grphiclly displyed in Figure 2, 35 40% of RR students met the Grde 2 spring end-of-yer reding benchmrk compred with 50 55% of students in the rndom smple. There were significnt differences in the pssing rtes on the Grde 2 end-of-yer benchmrk between the rndom nd RR students in ech cohorts, in fvor of rndom students (p<0.05). % Meeting G rde 2 Benchmr k (1) (2) (3) Cohort Rndom RR Figure 2. roportion of students chieving Grd e 2 end-of-yer benchm rk. After djusting for preexisting differences, the smll negtive effect sizes for the ssoc ition between prticiption in RR nd meeting the Grde 2 end-of-yer reding benchmrk were found for students in cohorts 1, 2, nd 3 (-0.30, -0.37, -0.47) ( See Append ix E, Tble E1b ). As such, the verge student in the rndom smple would hve higher chnces of pssing the Grde 2 end-of-yer benchmrk thn more th n 62% of the students in the RR smple (Springer, 2006; Becker, 2007). Grde 2 end-of-yer reding benchmrk by subgroup. Tble 4 presents the proporti on of students meeting Grde 2 end-of-yer benchmrk by sub group. rogrm Evlution Unit Reding Recovery Evlution 11

22 Tble 4 roportion of Students Achieving Grde 2 End-of-Yer Benchmrk by Subgroup Grde 2 Benchmrk Cohort 1 ( ) Cohort 2 ( ) Cohort 3 ( ) Subgroup % Rndom % RR % Rndom % RR % Rndom % RR Rce/Ethnicity Americn Indin Africn Americn Asin Americn Hispnic White Gender Femle Mle Specil Service Non-FARMS FARMS Not ESOL ESOL Not Specil Ed Specil Ed Note. RR= Reding Recovery. Results for Americn Indin nd subgroups with fewer thn five students re not reported. Overll, students in the rndom smple outperformed their RR counterprts in ech subgroup, except for cohort 1 specil eduction students. In ddition, the performnce of RR students in Grde 2 ws lowest for cohort 2 Asin Americn students; Hispnic students; nd students receiving specil eduction, FARMS, nd ESOL services cross cohorts. For these groups, the pssing rtes on the Grde 2 end-of-yer benchmrk for RR students, which vried by cohort, were mostly less thn 40%. Grde 2 spring TRC levels. The students in the RR smple strted Grde 2 with fll TRC levels tht were similr to or higher thn those of their grde-level peers (See Appendix F, Tble F1). Similr to trends observed for Grde 1, grphicl displys of the distribution of Grde 2 TRC-level dt illustrte tht, by spring of Grde 2, the distribution of Grde 2 spring TRC levels for RR students nd rndom smple peers were similr (See Appendix F, Figures F1 F3). As such, the spring Grde 2 TRC levels for the RR students re not clustered t the lower end of the distribution. While controlling for preexisting differences, ANCOVA nlyses found significnt differences in spring Grde 2 TRC levels between the two groups for students in cohorts 2 nd 3 in fvor of students in the rndom smple (See Appendix F, Tble F2). The negligible effect size for cohort 1 (0.09) confirms tht ny differences in the TRC levels between the two groups re not meningful in n eductionl setting. The negtive smll effect sizes for ssocition between prticiption in RR nd Grde 2 spring TRC (-0.42, -0.26) indicte tht the verge rndom smple students in cohorts 2 nd 3 would hve higher spring Grde 2 TRC level thn the mjority of students in the RR smple. Grde 2 TN/2 NCE scores. The expecttion is tht the mjority of the RR students would be t or bove the 50 th NCE on TN/2 nd tht there would be no significnt differences in performnce rogrm Evlution Unit Reding Recovery Evlution 12

23 between RR students nd students in the rndom smple cross cohorts. Contrry to the th expecttion, bout 20% 29% of the RR students cross cohorts were t or bove the 50 NCE on TN/2 ssessments compred with 45% 53% of their rndom smple grde-level peers (Figure 3). Follow-up nlyses were conducted to evlute the differences in proportions. Within ech cohort, significnt differences in proportions of students t or bove the 50 th NCE were found in fvor of the rndom smple students (p<0.05). Across cohorts, the negtive smll to lrge effect sizes (-0.40, -1.07, -0.59) confirmed tht the differences in the performnce between the two groups ws lrge enough to be meningful in n eductionl setting (See Appendix E, Tble E1c). Tht is, cross cohorts, the verge student in the rndom smple would hve higher NCE score thn the mjority of RR students (66%, 84%, nd 58%, respectively) % 50th NC E or higher Rndom RR 10 0 (1) (2) (3) Cohort Figure 3. roportion of rndom smple nd Reding Recovery students t or bov e the th TerrNov Second Edition 50 Ntionl Curve Equivlent. Grde 2 TN/2 NCE scores by subgroup. Tble 5 presents the prop ortion of stu dents t or bove th the 50 NCE for TN/2 ssessments by subgroup. rogrm Evlution Unit Reding Recovery Evlution 13

24 Tble 5 roportion of Students t or Above 50 th Ntionl Curve Equivlent for TerrNov Second Edition Assessments by Subgroup At or bove 50 th TN/2 NCE Cohort 1( ) Cohort 2( ) Cohort 3( ) Subgroup % Rndom % RR % Rndom % RR % Rndom % RR Rce/Ethnicity Africn Americn Asin Americn Hispnic White Gender Femle Mle Specil Services Non- FARMS FARMS Not ESOL ESOL Not Specil Ed Specil Ed Note. RR=Reding Recovery. -- indictes tht results for subgroups with less thn five students re not reported. A different edition of the TN/2 ssessment ws dministered ech yer (from 2004 to 2007). Depending on the cohort, the differences in performnce between the RR nd rndomly smpled peers cross the three cohorts were s follows: Asin Americns (10 25 percentge points); Africn Americns (22 33 percentge points); White students (30 47 percentge points); Hispnic students (17 29 percentge points). Groups with the lowest performnce on the TN/2 ssessment were the students receiving specil services nd Hispnic students (Tble 5). TN/2 scle scores. While the expecttion is tht the performnce on the TN/2 scle scores would not differ between the two groups, the men scle scores djusted for preexisting differences indicted the TN/2 scle scores for the rndom smples students were significntly higher thn scores for students in the RR smple (See Appendix F, Tble F2). The medium negtive effect sizes (0.46, -0.44, -0.37) for students in cohorts 1, 2, nd 3 lso confirm tht the performnce of RR students lgged behind tht of their rndom smple grde-level peers. Tht is, for ech cohort, the verge student in the rndom smple would hve higher TN/2 scle score thn the mjority of students in the RR smple. Grde 3 MSA proficiency levels. The expecttion is tht RR students will continue to perform s well s their grde-level peers in Grde 3. Figure 4 displys the proportions of students who were t or bove proficiency on the 2006 nd 2007 MSA. 6 6 No officil student-level MSA dt re vilble s of this writing for cohort 3, who were in Grde 2 during the school yer. rogrm Evlution Unit Reding Recovery Evlution 14