Choose an audience +

School Choice

Gold Standard Studies: Evaluating School Choice Programs

Gold Standard Studies: Evaluating School Choice Programs

The gold standard methodology in the social sciences is called “random assignment.” This method allows researchers to isolate the effects of vouchers or scholarships from other student characteristics. Students who apply for a voucher enter randomized lotteries to determine who will receive the voucher and who will remain in a public school; this allows researchers to track very similar “treatment” and “control” groups, just like in medical trials.  The following study citations and brief summaries are listed in chronological order, from most recent to oldest.

 

Study Citations & Findings

Jonathan N. Mills and Patrick J. Wolf, The Effects of the Louisiana Scholarship Program on Student Achievement After Two Years, School Choice Demonstration Project, Louisiana Scholarship Program Evaluation Report 1, Feb. 2016.

Findings: Louisiana—Students who applied to the Louisiana Scholarship Program in 2012­–13, won a school-level random lottery to receive a voucher, and attended a private school in 2012–13 and 2013–14 experienced a decrease in academic achievement compared to their peers who did not win the lottery and instead attended public schools. Lottery-winning voucher users had statistically significant large negative impacts on math achievement in their first year of the program that were less negative in their second year of participation. These students also had statistically significant negative impacts on English Language Arts achievement in their first year of the program; however, this negative impact dissipates to insignificance by their second year of participation. The authors said the results suggest that the negative impacts of the program may dissipate over time.

 

Atila Abdulkadiroglu, Parag A. Pathak, and Christopher R. Walters, “School Vouchers and Student Achievement: First-Year Evidence from the Louisiana Scholarship Program,” NBER Working Paper 21839, Dec. 2015.

Findings: Louisiana—Students in grades 3–8 who applied to the Louisiana Scholarship Program for the first time in 2012–13, won a school-level random lottery to receive a voucher, and attended a private school experienced a decrease in academic achievement compared to their first-time applicant peers who did not win the school-level random lottery and instead attended public schools. The authors suggested that the design of the LSP may attract a negatively-selected set of private schools struggling to maintain enrollment, noting that these findings imply that these schools also provided lower educational quality compared to the public schools.

 

Marianne Bitler, Thurston Domina, Emily Penner, and Hilary Hoynes, “Distributional Analysis in Educational Evaluation: A Case Study from the New York City Voucher Program,Journal of Research on Educational Effectiveness, July 2015.

Findings: New York, NY—While the authors found no visible effect on participants as a whole, they also divided students into quintiles based on student achievement and found no visible effects on academic outcomes in any quintile. They also replicated the racial group analyses of both Howell and Peterson and Krueger and Zhu using a different method of accounting for missing data. When they replicated Howell and Peterson’s statistical model, they found a positive effect for African American students; when they replicated Krueger and Zhu’s, they found no visible effect.

 

Matthew M. Chingos and Paul E. Peterson, “Experimentally Estimated Impacts of School Vouchers on College Enrollment and Degree Attainment,” Journal of Public Economics, Feb. 2015.

Findings: New York, NY—African American and Hispanic students offered vouchers to attend private elementary schools in 1997 attended college within five years of expected high school graduation at a rate 4 percentage points higher than the control group and obtained a bachelor’s degree at a rate 2.7 percentage points higher than the control group’s rate (11.7 percent vs. 9.0 percent, respectively). However, the overall voucher impact finding was null.

An earlier version of Chingos and Peterson’s study found African American students using vouchers to attend private school attended college within three years of expected high school graduation at a rate 8.7 percentage points higher than the control group and full-time college attendance rates were 8 percentage points higher, and the offer of a voucher to African American students resulted in their college attendance rate for a selective four-year college being more than double the control group’s rate (6.9 percent vs. 3.0 percent, respectively).

 

Patrick J. Wolf, Brian Kisida, Babette Gutmann, Michael Puma, Nada Eissa, and Lou Rizzo, “School Vouchers and Student Outcomes: Experimental Evidence from Washington, DC,” Journal of Policy Analysis and Management, Feb. 2013.

Findings: Washington D.C.—The students offered vouchers graduated from high school at a rate 12 percentage points higher (82 percent) than students in the control group (70 percent), an impact that was statistically significant at the highest level. Students in three of six subgroups tested showed significant reading gains due to the voucher offer after four or more years.

Overall, reading and math gains from the program were not statistically significant at the 95 percent confidence level, but overall reading gains were significant at the 90 percent confidence level. Parents remained more satisfied with their child’s school and viewed it as safer if offered a voucher, even though students had similar views of school satisfaction and safety whether in the treatment or control group.

 

Hui Jin, John Barnard, and Donald B. Rubin, “A Modified General Location Model for Noncompliance with Missing Data: Revisiting the New York City School Choice Scholarship Program using Principal Stratification,” Journal of Educational and Behavioral Statistics, Apr. 2010.

Findings: New York, NY—Using alternative methods, this study confirms the 2003 finding that, after one year, voucher students had math scores 5 percentile points higher than the control group.

 

Joshua M. Cowen, “School Choice as a Latent Variable: Estimating ‘Complier Average Causal Effect’ of Vouchers in Charlotte,” Policy Studies Journal, May 2008.

Findings: Charlotte, NC—After one year, voucher students had reading scores 8 percentile points higher than the control group and math scores 7 points higher.

 

Alan B. Krueger and Pei Zhu, “Another Look at the New York City School Voucher Experiment,” American Behavioral Scientist, Jan. 2004.

Findings: New York, NY—The voucher students had higher scores, but the results did not achieve statistical significance. Subsequent analysis has demonstrated that this occurred because the study used inappropriate research methods that violate the norms of the scientific community; if legitimate methods are used, the positive results for vouchers become significant.

 

John Barnard, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin, “Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American Statistical Association, June 2003.

Findings: New York, NY—After one year, voucher students had math scores 5 percentile points higher than the control group.

 

William G. Howell and Paul E. Peterson, The Education Gap: Vouchers and Urban Schools, Brookings Institution, 2002, revised 2006.

Findings: New York, NY—After three years, African American voucher students had combined reading and math scores 9 percentile points higher than the control group.

 

William G. Howell and Paul E. Peterson, The Education Gap: Vouchers and Urban Schools, Brookings Institution, 2002, revised 2006.

Findings: Washington D.C.—After two years, African American voucher students had combined reading and math scores 9 percentile points higher than the control group, and all voucher students had combined reading and math scores 7.5 percentile points higher than the control group.

 

William G. Howell and Paul E. Peterson, The Education Gap: Vouchers and Urban Schools, Brookings Institution, 2002, revised 2006.

Findings: Dayton, OH—After two years, African American voucher students had combined reading and math scores 6.5 percentile points higher than the control group.

 

Jay P. Greene, “Vouchers in Charlotte,” Education Next, Summer 2001.

Findings: Charlotte, NC—After one year, voucher students had combined reading and math scores 6 percentile points higher than the control group.

 

Jay P. Greene, Paul E. Peterson, and Jiangtao Du, “School Choice in Milwaukee: A Randomized Experiment,” in Learning From School Choice, ed. Paul Peterson and Bryan Hassel, Brookings Institution, 1998, pp.335-56.

Findings: Milwaukee, WI—After four years, voucher students had reading scores 6 Normal Curve Equivalent (NCE) points higher than the control group, and math scores 11 points higher. NCE points are similar to percentile points.

 

Cecilia E. Rouse, “Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program,” Quarterly Journal of Economics, May 1998.

Findings: Milwaukee, WI—After four years, voucher students had math scores 8 NCE points higher than the control group. NCE points are similar to percentile points.

 

Suggested Citation
“Gold Standard Studies: Evaluating School Choice Programs,” Friedman Foundation for Educational Choice, last modified Feb. 24, 2016, http://www.edchoice.org/school-choice/gold-standard-studies.

Questions about Educational Choice?

Choose your path.

Receive Educational Choice Updates Straight to Your Inbox.