The Five Things You Should Know about the New D.C. Voucher Test Score Study

This week, the Institute for Education Sciences released a new report on the performance of the District of Columbia’s Opportunity Scholarship Program. This is the second report of the second evaluation of the program.

To cut to the chase, the topline findings of the study are not encouraging. Two years after application, voucher lottery winners scored on average 8 percentage points lower than their lottery losing peers in Math, a statistically significant difference. In reading, lottery winners scored lower by 3 percentage points but that difference was not statistically significant.

But there is more to this study than meets the eye.  Let’s walk through five quick takeaways:

1.  D.C. is a unique case. Washington, D.C., is one of the most choice-rich environments in the country. In fact, 43 percent of the “control” group (lottery losers) attended public charter schools. D.C. charter schools have consistently shown high levels of performance, so comparing voucher students to charter students sets a pretty high bar. As the authors themselves write:

“It is important to note that students in both the treatment and the control groups scored higher on the tests two years later than they did at the time of application. The impacts were negative because the gains in test scores for the treatment group were smaller than the gains in test scores for the control group. An analogy is to a footrace—all students are running forward but the control group students are running faster.” (page 20)

What’s more, the voucher program has been a political football and has been at the brink of closure multiple times since its inception. This affects who wants to participate in the program and to what degree they want to participate. Also, while not quantified in the study, the authors do cite a statistic that, in 2012, 75 percent of public school students in the District did not attend their neighborhood school. So even in the 47 percent of the control group who are listed as “traditional public school” students, many are probably in public schools of choice as well.

I’ll let the authors summarize how they think all of this affects their study:

“The combination of elements—a program whose funding and support has shifted over time at the federal level, operating within a city that offers ample options for parents to choose schools—makes findings from this evaluation challenging to generalize to other settings, such as voucher programs operated statewide or in settings that currently have limited choice options.” (From page xvii of the Executive Summary)

 

2.  There is some good news. The evaluation did find extremely large benefits from the program when it comes to student and parent reports of school safety. The 16 percentage point difference for parents and 11.6 percentage point difference for students were both statistically significant and meaningful. This is not nothing. Students feeling safe, and their parents feeling that they are safe, are essential to building a good school culture. I can’t speak for anyone else, but given the option between a school that raised tests scores but that I thought was unsafe and a school that didn’t raise test scores but I felt was safe, I’d pick the latter every day of the week and twice on Sunday.

3.  The Opportunity Scholarship Program funds students at rates far less than their traditional public and charter school peers. Vouchers are worth $8,000 for K–8 students and $12,000 for 9–12 students. According to Governing magazine, Washington, D.C.’s traditional public schools spend, on average, $19,396 per student. According to the DC Fiscal Policy Institute, D.C. charters spend, on average, $14,639 per student. Promoting student learning gains, but gains at a lesser rate than their dramatically better-resourced peers, is again not something to denigrate. It would be interesting to see how well those schools could do if they were funded at the same level.

4.  Attainment results matter too. For those of you with a long enough memory, the first evaluation of D.C.’s program found no statistically significant gains in reading or math but large and significant gains in graduation rates. In a recent study, Matt Chingos of the Urban Institute found no statistically significant differences in college enrollment rates between lottery winners and losers who applied in 2004 and 2005. There is a lot going on in D.C., and recent graduation rate scandals have rocked the traditional public system, so it will be interesting to see what the longer term effects of the program are.

5.  What if parents don’t like what we think they like? As someone who has conducted and consumed school choice research for some time now, I think that there is a growing preponderance of evidence that parents simply don’t value test scores as much as researchers and policy wonks do. This shows up in the surveys of participating parents that we have conducted. If parents are choosing for reasons other than test scores, we shouldn’t be surprised if their choices don’t lead to higher test scores.

Almost all school voucher programs show some dips in student performance in the first year and then a rebound over time. This is what we have been seeing in Louisiana, and it will be interesting to see if the trend holds in D.C. as well. We’ll have to wait and see.

But, more than anything (and I’m saying this as a person who does school choice research for a living), research can only tell us so much about the value of a school choice program. Giving parents the agency to choose where their children go to school and allowing for a more pluralistic and open system of education to flourish is impossible to quantify. We use reading and math scores to measure programs because reading and math are easy to measure. They are a limited view of what we want out of education. We shouldn’t lose sight of that.