Members of the EdChoice team discuss the recent publication, The 123s of School Choice, an in-depth review of the available research on private school choice programs in America.
Mike McShane: Hello and welcome back to another edition of EdChoice Chats. This is Mike McShane, Director of National Research at EdChoice, and today we are going to be talking about a new publication or really a new version of an old publication, something we’ve been working on for many years now, TThe 123s of School Choice. I’d like to think, especially of us on our research team, we have a few of these kind of tent pole products that we put out every year. The ABCs of School Choice is obviously probably the best known of that, which is the physician’s desk reference of school choice programs across the country.
If you ever happen to be curious, does my state have a school choice program, or how many school choice programs exist across the country, where are they, how many kids are enrolled, how much money do they get, all of that information is contained in The ABCs of School Choice. Building on the success of The ABCs of School Choice, we created The 123s of School Choice, which is a physician’s desk reference or compendium of all of the research that has been done on private school choice programs. I will let the leader of our team working on this, Marty Lueken, give more of the gory details.
But I think we crossed the 190 study mark this year, where I think there were more than 190 studies that are summarized in this report. I should say I’m joined today by my colleagues, Marty Lueken and John Kristof. We should say we were going to be joined at one point by our colleague Colyn Ritter, who also contributed to this report, but then there was four of us talking. This ended up working out. But shout out to Colyn, did a lot of great work on this, and we appreciate it.
But Marty, since you were running the bakery on this one, maybe you could just give folks the big picture, what is the purpose of this report? Maybe a little bit of history, how it came to be, but what are we trying to accomplish with The 123s of School Choice?
Marty Lueken: Well, thanks, Mike. And again, thanks to the whole team. This really was a team effort, Colyn and John, you guys did a lot of work on this. The purpose is pretty simple. Folks want to know if current school choice programs work. People in states, such as policymakers who are in states that are considering introducing school choice programs in their states, they want to know how the programs might affect various kinds of people. Really this publication came about… Well, going back to I think it was around 2016 maybe when Greg Forrester, who is one of our fellows, he wrote a win-win solution.
At the time, we were Freedom Foundation. That was the first synthesis of research at that time that was produced, at least in a really nice, easily digestible manner for a broad audience. The 123s is an evolution of that. In this publication, we look at a variety of outcomes. We look at the effects of choice programs, education savings accounts, voucher programs, tax credit, scholarship programs, what effects did they have on program participant test scores, how do they affect attainment, which is a student’s likelihood of graduating from high school or enrolling in college and assisting in college.
They look at parent satisfaction, public school students test scores, civic values. They look at racial and athletic integration and fiscal effects. This addition, we have one new outcome, school safety, which is something that parents really care about. The 123s, it’s basically a synthesis of what the research says. As you pointed out, there are now almost 190 studies of school choice programs on any of these outcomes.
Mike McShane: We didn’t quite cross the 190 threshold. We are approaching 190.
Marty Lueken: We’re so close. I think it’s 187.
Mike McShane: Now, it’s important I think to highlight at least for a moment, dig into something that you just said there, which is what The 123s is and what The 123s isn’t. Because sometimes The 123s can be used as kind of a battering ram and sometimes it’s used as a punching bag. Perhaps better put, it is misused as either a battering ram or a punching bag. I think it might make sense, Marty, if you could just walk through what it isn’t in the sense that it’s not a meta-analysis, it’s not trying to make definitive statements, but what makes this different from maybe other efforts to synthesize school choice research? What does The 123s tell us that that stuff does and what does that stuff tell us that The 123s doesn’t?
Marty Lueken: Yeah, no, that’s a very good point, Mike. It is not like a meta-analysis. The meta-analysis is basically a study of studies, a statistical analysis, which it collates all of these effect sizes from a research on for some particular outcome, and it tries to boil that down to one number. And then if it’s positive, it might be good or bad. And then negative, it’s the other way around. What we try to do is present all of the research where the methods have reached a certain level of quality and we just try to lay it out there.
Basically we say, “Overall, what does it show? Are the findings overall positive? Are they overall negative? Or do they even detect an effect if sometimes studies may find both positive and negative outcomes?” We will record that as well. I think it’s more to give a broad overall general picture of, one, what studies look at, what are the outcomes that we have information on, and then two, overall broadly, how is choice doing what those studies say.
Mike McShane: John Kristof, if the goal of this report is to say what the effects of school choice are, I could set you up with the tough one or potentially an easy one saying, so what are the effects of school choice? But maybe this is the section of the conversation where we get to what are the effects of school choice. I don’t know if we want to start at the top. It doesn’t maybe make sense to go through every single study or everything that’s there, but generally speaking.
I mean, so one of the ones that I think people care a lot about is effects on students who participate. The one way to measure that is through test scores. What do we know about the participant effects? Students who participate in voucher programs. They take standardized tests. What do they tell us?
John Kristof: This is maybe a good one to start out with because this is the effect that gets studied and maybe discussed the most. We talk about that a little bit in our preface to the document. When you look at the results, and we have a pretty little table that you can see in the report, if you spend enough time on social media, you’ve probably seen this circulated around at some point if you’re in choice circles on social media, we basically just identify of all the random assignment studies, and I’ll come back to that in a second, that have studied the effect of school choice participation on test score outcomes, the number of studies that find positive results on students significantly exceed the number of studies that find some kind of negative result.
What do I mean by random assignment studies? We talk about this pretty in depth in the report. This is just for ease of use. Because there have been so many studies on test score outcomes for students participating in school choice programs, we have the ability to, if we’re going to synthesize the research, present the best research. Any prof in a social science program will tell you that a random assignment study is more or less the gold standard of social science research because it is the best way to isolate what is the effect of the treatment in this experiment, or in other words, in this particular case, what is the effect just of the school choice participation as opposed to any other factors?
You do that by basically being able to randomize whether a student gets a voucher in these cases or not. There have been some cases in some states where if there’s an oversubscription or an excess of demand for the program and the winners for the spots available are randomly chosen, that randomization allows you to… Or at least on average, the randomization means that we can more or less assume that both groups are essentially equal on average, aside from whether they participate in the school choice program or not. They’re just as likely to want to participate in a school choice program.
That’s not a factor, for example. The strong majority of these studies find null to positive effects of school choice research. There have been a couple studies that have found some kind of negative results. There was mixed results in the Milwaukee program in 2008, which competes with older research in that area. For some groups, there was a positive effect. In some groups, there’s a negative effect. And then there’s the, dare I say, notorious Louisiana program where some negative effects have been found. There’s been a lot of discussion as to what’s going on with the Louisiana program specifically as to why that might be the case.
Now, again, this is not a meta-analysis, although some people have done meta-analysis. Pat Wolf at the University of Arkansas, along with some co-authors, has produced a meta-analysis on this question where they worked really hard to find an average effect of what participation in school choice programs were, found a positive effect. But this synthesis is helpful because this is a culmination of all of the research that is especially relevant, and then we make some other references to some studies that you might see tossed around sometimes as well. This isn’t a one pager.
I mean, we do have a one-page table, but we are not trying to whittle down the studies that we think are helpful or were most important. We think this is good research that has been done on this question of what effect a school choice have on test score outcomes. We find that generally there are positive results in this area. With some weird ones, we’re not hiding those. We’re not creating our own guide. We’re saying this is a standard by which you would want to figure out your own path learning about school choice research, and we just present it all.
Mike McShane: It’s important to note, Marty, there have been some recent studies of participant effects of school choice programs that we do not include in our summary tables. We do have some. They’re included in a narrative version where we talk through them. But it might be worth just helping our listeners for a minute or two here some of these more recent studies.
I know sometimes a criticism of this report is like, well, why aren’t they included? Could you give us the brief lowdown? John already touched on it about the value of random assignment studies and others, but broadly speaking, why those studies? If someone wants to skip the podcast at this point to say like, “Hey, why isn’t XYZ study in here,” why isn’t it?
Marty Lueken: There are some studies that have been done on programs in, for example, Indiana, Ohio, also Milwaukee too, which do not use random assignment. These are studies that use matching techniques. A matching study is basically you take a student that’s participating in the program, and then you try to find based on a certain set of information its virtual twin, so to speak. You try to find someone in the public school system which looks like this individual. They might be in the same district. They might have a similar or same household income level. They both are ELL, or they both are not, et cetera, et cetera.
And then you have your treated group and then this matched comparison group. You just compare the outcomes of the two groups and then get your effect size. Now, the issue with that is that there are also a lot of things that cannot be measured. We can only measure on things that we can observe and that we can measure, but there are all sorts of things that we can’t measure as well. For example, someone’s level of motivation is one classic example. The issue in the context of choice is that what you have is one group who is making the choice, who’s selecting into this program, and then you have this other group that on paper looks like them, but they’re choosing not to opt in, not to select.
There’s something going on likely that we can observe. For example, they may have issues with bullying at their former schools, and so some students might be opting into a choice program because of that. That’s hard to measure. That’s usually not tracked. We can’t match on that. Right there, There’s an example of how these groups might be different. That introduces bias into these estimates. These research methods, they don’t give as clean of an estimate as a random assignment study does.
Mike McShane: That’s very helpful. Yes. Look, ultimately, when it comes to making decisions about inclusion and exclusion and research quality, there are lots of judgment calls that have to be made, like what makes higher quality versus lower quality research, more policy relevant research and others. We do our best as possible in this report to line up why we made the decisions that we did. We went so far as to even include descriptions of the studies that are not included and the reasons around here. I think it’s important to just highlight, we did our best to be transparent in this. We understand some people might.
Your mileage may vary, as they say, but I think we’ve picked a very clear, honest, and defensible way of including what we are including. Moving on from that, and I think in some ways people can justly use this report as a myth buster, but one of the most persistent myths I think out there about private school choice programs is that they hurt the students left behind in public schools. Now, obviously this is a super important question. If we had reason to believe students leaving with vouchers caused public schools to be much worse off, it’s not 100% clear to me that that would necessarily be a bad thing because I think those kids have a right to a quality education.
If they’re not getting it, we should do our level best to get them what they can. But I think it’s a perfectly reasonable thing for folks to be concerned about. John Kristof, can you give folks a sketch of the research literature on, I guess, what we refer to as the competitive effects, but is really functionally what happens to kids who are left behind in public schools when voucher programs or other private school choice programs are introduced?
John Kristof: Right. Milton Friedman, who quite arguably the earliest advocate for school vouchers as we know them, and therefore the private school choice movement in general, one of his original arguments for school choice is that it would make public school system better through these pressures then. There’s actually consequences to how good your service to your community is, how good the educational services to your students are, both in an academic sense and in a social and emotional sense as well.
Milton Friedman, and we are more or less his intellectual legacy, we also believe that public schools will be better off in the long run when school choice exists and some students switch the school type that they are going to. The evidence for this being the case is I think, pretty outstanding, pretty strongly positive. The numbers are that there have been 29 studies that have done competitive effects research on private school choice, and 26 of them have found positive results. It’s pretty significant. This is been as the most recent studies in 2022 for the Ohio Educational Choice Scholarship Program, and this research goes all the way back to 2001 with Florida’s Opportunity Scholarship Program.
This research has been going on for a while and we’ve been finding pretty consistently positive results throughout the time. There’s one quote in particular that I think we’ve included in the report from one meta-analysis of these competitive effect studies, which again, we’re not meta-analysis. This is a synthesis publication, but there has been a meta-analysis done of this. They say there essentially is little empirical argument here that public school students are going to be harmed. Little justification to use that as a reason to dismiss private school choice programs.
The evidence here is pretty strong, regardless of your perspective, regardless of the method type that you do, regardless of where you’re studying it, when you’re studying it, how old the program is. The effects here are pretty strong.
Mike McShane: Now, I want people to check out the actual reports, so we’re not going to go line by line through every finding that we have in there. But Marty, I did want to throw out, we included a new category this year. Could you explain to people what we included, why we included it, and what we found or what we synthesized?
Marty Lueken: We included one new outcome, school safety and climate. This is a category… Well, first of all, this is an outcome, which as we know from a lot of our polling and survey work, this is one of the things that parents not only care about, but they tend to rank highly among the various different factors that go into their decisions for school making. This tends to rank the top three items of things that they care about when it comes to their kids’ schooling. These are studies that measure things various like safety related issues, like student bullying, physical conflict in schools, gang activities, drug related or alcohol related problems, discipline issues and so forth.
Overall, there have been eight studies that we identified and included in this edition of The 123s, and every single study has detected positive effects. Not one study detected any negative effect when it comes to school choice’s impact on school safety. I’d also like to mention too that in addition to this new outcome, we do have a new section too. 123s, this is one of many research syntheses that have been conducted going back about 20 or more years. We’ve identified about almost 30 other research synthesis on private school choice research.
We tried to also collate those, and we’ve included and tried to summarize those and The 123s as well. Definitely want our readers and audience to check that out too when they get a chance.
Mike McShane: Yes, that’s very important. We’ve been spending some time here talking about two of the categories that rely on standardized test scores. It’s important for folks to recognize that obviously there’s much more included that in The 123s because we think that there’s a lot more happening in schools than is measured by standardized test scores. It’s the researcher’s dilemma that standardized test scores are usually easy to get your hands on. They’re administered a lot. If you want to make comparisons, especially if you’ve got different groups of people but have all taken the same test, there’s a very natural temptation to make them the sum of all things, which they aren’t.
We’re very happy to include all of these. But maybe I want to close by asking each of you a big picture question, which is obviously this is a massive report and kudos to you for all of the hard work that you all have put into this, synthesizing this, going through, making all the difficult decisions. I know there’ve been lots of agonizing, should we include this study or not include this study? Or how should we classify different findings? What are some hard and fast rules that we can make to try and make this as fair and transparent as possible?
But if you’re talking to someone and you only had one takeaway, that if people were just flipping through and you hoped that they captured one thing from this entire project, John, I may throw it to you first, but what is the one thing you hope people walk away with from The 123s?
John Kristof: I hope the main impression that people can get that I think is pretty uncontroversial is that school choice research has a very rich history. If you’re interested in educational choice, honestly, whether you have presuppositions for it or against it, I hope that you can look at this study and appreciate how much work has been done to try to understand what effect it has so far. School choice has a lot of momentum behind it right now. There’s been a lot of exciting legislation passed recently, but that doesn’t mean necessarily that it’s the Wild West right now. We have no idea what’s going to happen. There is a lot of really strong research that has been done understanding the effects in a lot of different ways.
I’m going to cheat and do a second thing that I hope people recognize. This is just hammering home something that we’ve talked about a little bit, and that’s that asking whether something works assumes that you understand what you’re trying to achieve. With all the experience of polling work that we’ve done here and we find all of the different things that people want out of an educational choice, the system, everything that people want out of their schools for their kids, you cannot just end the discussion at test score results. Because if you do that, you are valuing most the things that people value least or nearly leased in an educational system.
I hope that people reading this can get a sense for all the different ways that people have tried to measure success or at least outcomes, things to consider from an education reform. Honestly, the education reform movement, school choice or otherwise, has significantly shifted what it has cared about in the decades and decades that it’s existed, going all the way back to the 1970s honestly. What we cared about when the nappe test was created is different from what we care about now, left, right, and center, or at least what we prioritize. If that’s the case, we should also diversify to the best that we can what we study as well.
Hopefully this can give some people some ideas about what that can look like. Maybe there’s some researchers who are taking a look at this for the first time and have an idea for, I’ve seen this methodology or data collection technique used here, and I think it could work really well for this research outcome here. We would love that. We’d also love for this to be inspiring for more research in the future.
Mike McShane: Well, Marty, John bought you some time there, but the question is to you, what’s the one takeaway you hope people take away from this project?
Marty Lueken: What really stands out to me is how few studies we’ve found that have detected a negative effect of school choice. The reason is that these choice programs are funded at just a very small fraction of what students in the alternative, in the public school setting are funded at. Choice programs receive on average about one-third of the per people dollars that students generate for district schools. Given that funding difference, one might expect to find a lot more negative effects.
Now, granted we do in some studies, but it’s a really small minority. Choice still has generated in the research a lot of positivity. I think that that’s an incredible thing. I think that itself really speaks to a lot of concern that folks have about choice programs in particular when we hear a lot of claims that these programs harm students that remain in public schools. I think there’s just a lot of positive that we can see in this research.
Mike McShane: Look, I will add my bit of host editorializing here, building I think off something John said, but touching on what Marty said too. From our other research that we have, which you can find on EdChoice’s website, www.edchoice.org, we recognized that people choose schools for lots of different reasons, and therefore to ask and then attempt to answer a question like, does school choice “work,” we have to ask like, well, what do you want it to do? What do you want from schools? What does it mean for schools to work? What do we want from schools? And then we back out. Well, then how do you measure that and how do you report that and how do you compare that?
I think one of the great values of this report is that it doesn’t try to pick one definitive answer to that, right? We don’t say, well, the one that matters for working is the test scores or something else, the safety or the competitive effects or any of those things. We present all of it. We imagine, I think, different people could read this report differently, highlighting the stuff that they care about. There are folks who really, really care about school safety, so they might flip immediately to that. That’s where they start the report, and then the other stuff is interesting. There are maybe people who really place a lot of value in test scores and the academics that they measure.
They might start there and go somewhere else. Folks who really care about the public school system and how these programs intersect with them could start there. It’s almost like a choose your own adventure novel. You can find the thing that matters most to you. We don’t try to say this is the one thing, this is the one yard stick that school choice should be measured by. There are lots of different people who choose for lots of different reasons. We live in a big diverse country where different people want different things out of schools.
I think the beauty of this publication is that it allows for all of those people to learn about things that they care about, understand the studies, and go through the bibliography that’s in it and find the actual studies themselves. I think it’s a wonderful resource. I think that you all and the rest of the team who put it together, and we have to give a shout-out to our communications folks and others who lay it out and make it look beautiful and push it out into people’s social media feeds and into their email inboxes. Kudos to all of them for everything that they’ve done. But I think it’s a really valuable resource that people can get a lot out of, provided that it is used properly and not misused.
Well, John, Marty, it was a pleasure as always. Everyone can head to our website, www.edchoice.org, to see The 123s in all of its glory. We’ve been done SlideShare versions of it where it’s really easy to flip through. We also have a more text-based version where you can get all of the reasons why we did the things that we did in our own ruminations on what all of the findings tell us. I mean, without further ado, you all should do that now. Stop listening, hit pause, hit end, and go to it. I’m going to stop talking. I hope you all enjoy reading that.
Thanks, as always, to Jacob Vinson, our wonderful podcast producer who’s going to put all of this together, and I look forward to chatting with all of you again on another edition of EdChoice Chats.