A few authors of our annual report, The 123s of School Choice, discuss what you’ll find in this year’s edition. Click here to view the fully digital report.
Mike McShane: All right. Hello and welcome back to another edition of EdChoice Chats. This is Mike McShane, director of national research at EdChoice. And it’s great to be with all of you again. Today, I am joined on the line by my friends and colleagues, Drew Catt, Marty Lueken, and an honorary member of the research team today, Jordan Zackery. Jordan, welcome to the nerdom. Your glasses and pocket protector will be issued to you shortly, but it’s great to be with all of you today. So, today on the podcast, we are talking about a new document that we put out, or frankly, a new version of an old document, a document that we’ve been working on for some time now, called The 123s of School Choice, subtitled, “What the research says about private school choice programs in America.” And so obviously, our work on the research team, we’re frequently asked the question, “What does the research say about school choice?”
And I think people want us to be able to answer that in one or two sentences. And as we will get into in the conversation today, it is slightly more complicated than that. All the different ways you can wack that pinata of questions about how school choice works. But maybe to give the kind of 30,000-foot overview of The 123s of School Choice, I’m going to turn it over to Drew Catt, and then we’ll get into kind of, this is a document that we update every year. And so, the new updates have just come out, and we’ll talk about what that looks like. And we’ll talk about how this thing gets used. But Drew, maybe to start with you, that kind of 30,000-foot look, what is The 123s of School Choice?
Drew Catt: Yeah, thanks Mike. So, the way that I like to describe it is, especially, if I’m at a booth at a conference, it’s been a while since that has happened. Can’t wait to see some of you all out there in person again, here in the near future. But yeah, so I would say, “Hey, have you heard of our ABCs of School Choice? That’s like our main flagship publication. It’s the comprehensive guide of all the school choice programs in America. Well, this is The 123s of School Choice. It’s more or less the compendium of research on private school choice. And it’s not just the voucher programs, the tax-credit scholarship, the ESA programs, but it’s also looking at all forms of private school choice, including the privately-funded school choice.”
Because, when it comes down to the research, the way that the funding works, it doesn’t matter. It’s what actually happens. So, yeah. This is the compendium of all the research. Well, most of the research on school choice as we’ll get into. We do not include every single category of research in here yet. But yeah, so I guess it’s a collection of research that, Marty Lueken has done the humans work and putting it all together and classifying everything. But yeah. It’s a lot of fun to be able to have and to pick up and to see use.
Mike McShane: Yeah, for sure. So, as Drew said, compendium of research on school choice. So, we get asked that question, what does the research say on school choice? And there’s a lot of different ways of answering that question. So, The 123s of School Choice goes through a couple of different potential research questions. So, we look at program participant test scores. So, that’s like students who receive a voucher or a tax-credit scholarship or an ESA, and how they perform compared to other students who don’t. Program participant attainment, so that’s looking at how they graduate from high school, matriculate into college, etc. Parent satisfaction, public school students test scores. Obviously a big question we’re always asked is, “What happens to kids who were left behind in public schools when people leave after participating in school choice programs?” Looking at civic values and practices, racial and ethnic integration and fiscal effects.
So now Marty, that’s like a whole bunch of stuff, right? Like a lot of different questions. So, given that as Drew correctly said, you are the quarterback of this project. How did you go about actually collecting all of the studies that became part of this compendium?
Marty Lueken: Sure. So this is really a continuation of a series of reports that we called “Win-Win.” Where Greig Forester, he’s one of our fellows. He did these early summaries of research on school choice. So, we started from there and we recruited Hanover to do a backfill. So, it’s a private research firm that we often collaborate with and we’ll do work together with, and did this backfilling just to make sure we didn’t miss anything. And then we had our first edition of 123s in 2019. And every year, since we basically continue doing these searches for research on these seven different outcomes, which he mentioned, and we have a process where we review the titles, the abstracts and all that, and all the details we have listed and discussed in this resource.
So, if you’re interested in those details, you can refer to them. So yeah. Then we review and determine whether these studies have detected any positive effects, any negative effects, or in some cases, just don’t have enough information to detect an effect at all.
Mike McShane: And so, by my count, it looks like, I think we included 169 different studies this year? Am I right? Is that the right number?
Marty Lueken: Yep, that’s correct. Yeah.
Mike McShane: Wow. So, across all these different areas. So, maybe before we dive into those, Jordan, I might throw it to you for a second. So, Jordan you work out in the states with legislators, with coalition partners, with all of those folks. I’m curious, what kind of questions do you get about school choice research? Like what do people want to know?
Jordan Zackery: Sure. Well, first I just want to say The 123s are one of the most desired resources that we have when it comes to legislators and coalition members alike, and yeah, a big thing that gets brought up are the student attainment.
They want to know not just program participant test scores, but they are curious about public school counterparts as well, which that information is all in The 123s. A little more specific when I’m talking to parents, they do care about test scores because they want to know that school choice will benefit their students. But they’re also interested in sections of The 123s like parent satisfaction. So, that is really big with coalition members being able to provide that data to parents, so that they know that school choice is right for them as a parent and right for their students.
When it comes to legislators, they really do love the test score information and they love the fiscal effects. They like hearing that school choice in many states has been great for taxpayers. That’s something that’s really important to them. Just thinking back to this session already, Rep. Glenn Cordelli, used 123s during the House education committee in New Hampshire, to talk about the test scores, to talk about the fiscal benefits of school choice. Senator Amy Sinclair, she presented information from The 123s on the Senate floor to make sure that legislators knew that one, taxes would not increase as a result of school choice, and two, that public school students would not see harm of their performance because of school choice. So, The 123s have really encapsulated a really big chunk of the research that legislators and coalition members are looking for when they’re trying to talk about the issue.
Mike McShane: Okay. So, let’s dig into some of these numbers, and obviously folks, you should go to our website, www.edchoice.org, to actually get all of the gory details. We’re not going to try and talk through all 169 studies that were here. But maybe to hit some of these big questions. Drew, I might throw it to you first. So as Jordan brought up, a lot of people are interested in test scores. So, I will ask you the question, “Do students who participate in school voucher programs, or private school choice programs, have higher test scores than similarly situated students who do not?”
Drew Catt: Yeah, thanks Mike. On aggregate the answer is yes. Wholehearted yes. This is one of the areas that we get really nerdy, I guess if you would say, in the research world. And that we focus entirely on the random assignment studies, that is students could be randomized to one group or another due to the way that the programs were set up. So, these are the more “scientifically rigorous studies” which in the research world, all research can be a good research. So, I won’t split those hairs. That’s a different podcast. But of the 17 random assignment studies conducted, 11 have found positive outcomes for either everyone, which research where we call it the full sample, or at least one sub sample of students studied. Whether that’s just mostly the elementary school students, whether that’s just males, whether that’s Hispanic females, and then four of the studies found no visible effect for any of the groups of students, and three, found negative outcomes for all or some students.
However, two of those negative effects studies, did focus on the same program whereas, one of the others that showed negative effects, also showed positive effects for sub students and negative effects for some other students. So, it was kind of a mixed result for that one.
Mike McShane: Yeah. Sorry, Drew, I gave you probably the hardest one to try and explain. A lot of the other cases as those of you who check out The 123s of School Choice, will see very clearly there’s a much less ambiguity in the other questions, but that was a tough one to talk through, but well done.
Drew Catt: Thanks. I was hoping that you’d point that one to the other guy on the call with a Ph.D. besides yourself, Marty Lueken, but hey, more than happy to swing the bat.
Mike McShane: On the other side of this, I’m going to throw it up to Marty, because I think one of the other questions lots of folks brought up, a big concern that people have is, what happens to the children who were left behind in public schools?
And I think that this is a perfectly fair question. I think that it should matter to us, that if some percentage of students leave because they participated in a private school choice program, but all of the kids who were left behind in public schools were much worse off as a result of that, that’s something that we need to take into consideration in the calculus of saying whether these programs are good ideas or bad ideas. Marty Lueken, Doctor, as Drew Catt brought up. Dr. Marty Lueken, I would like to know what does the research say about students who are left behind in public schools?
Marty Lueken: This is absolutely an important question that we get asked about often. And there are a lot of concerns. So basically, the idea being that if you have students leaving for these programs, you’re going to have cream skimming going on, right? You’re going to have the siphoning of dollars from the public school system, which are partially tied to enrollment counts, as public schools are going to lose out on dollars. And therefore, the students who remain are going to be made worse off somehow. Fortunately, there’s a quite a large body of research that examines this very question. What happens to students’ academic achievement? And there’ve been over two dozen studies. So, 27 studies have looked at the impact of choice programs on students’ test scores. And of those 27, 25 have found that public school students are affected positively, that they experience positive gains. Now these are modest gains, they’re small, but they’re not negative, right? They’re positive. None of the studies found no visible effect. And then there was one study that found the negative effect. And interestingly, that study that found a negative effect was done in Florida.
And it looked at what happened after a choice program closed. There have been a handful of other studies that have looked also at programs in Florida, and all those other studies have found positive effects. Now, these effects sizes are small, they’re modest, but they’re still positive. And more recently, there has been another study on Florida, which actually looked at the impact of scaling up its tax credit-scholarship program. And that study, which was done by David Figlio and a couple other researchers, it found that as Florida scaled up its program, again, students who remained in public schools, not only experienced increases in their test scores, but they also looked at some behavioral effects. So, they looked at the impact of scaling up on absenteeism and suspensions. And they found that students who remain in public schools, they have less absenteeism, fewer suspensions, more test scores, and these effects tend to be stronger among students from lower socioeconomic backgrounds as well.
Mike McShane: Well, you help preempt my next question, so obviously we have been doing The 123s of School Choice for some time now, and you mentioned the Florida scaling up study is new this year. So I would be interested to know, especially for folks that maybe Jordan has given them a copy of The 123s before, what is different in this year’s iteration of it than in previous versions of The 123s of School Choice?
Marty Lueken: Not much. There have been some papers, the working papers that were either updated with new data or they might’ve been published in an academic journal. Those changes didn’t really affect any of the material results or the qualitative results. There have been some new studies since then, for example, there was one study done on attainment, looking at attainment effects on Indiana’s voucher program. And then there have been a couple fiscal analyses or papers. One of them was done by me looking at the fiscal effects of these programs on test scores. So, in terms of anything we’re doing methodologically, nothing there has changed. The only thing that really has changed has been, some new research and updates of current studies that we’ve already included before.
Mike McShane: So, Jordan, I’m sort of putting you in the hot seat here, given that the rest of us are on the call with you. But I want to preempt this question by saying, regardless of how you answer it, you won’t hurt our feelings. Okay. I just want to put that out there, so you can feel comfortable answering this question, but does anyone care? Like we’re putting out all these studies, 169 different studies of school choice as both Drew and Marty have summarized, the lion’s share of them show positive results for students across a variety of indicators.
And yet it seems to me, if I open up Twitter, or even mainstream news coverage or others, the conversation often seems to be completely ignorant of these facts. So, is it just that people, they hear research and their eyes glaze over? Is it that we don’t always do a good job explaining the research to people? Do people not care? How do we make more people kind of aware that this is the state of the research on this issue?
Jordan Zackery: Yeah, Mike, I think in some ways you kind of answered the question yourself. It’s about how it’s displayed. People do care, and also legislators care. I know Twitter does have a really big influence on public perception and social media really affects how people think. And that’s why it’s important that the research not only is put on all those mediums, just like the content from people who don’t use research, who don’t trust the numbers.
The research needs to be put in those mediums, and it needs to be put in a way that’s digestible to people. And I actually think EdChoice does a really good job at doing that. I might be biased because I work for EdChoice, but my feedback I get from our state partners is that a lot of our research is easy to read, easy to follow, and we’re improving on that. We have a really great comms team that helps make some of our research digestible. And we have you guys who do a great job with that too, but yes, people do still care about the numbers. Legislators really do care about the numbers, and equipping them with the tools, equipping them with our resources, so that they can talk about it to their constituents is really important. And I think you’re seeing it this year.
There’s a wave of school choice programs being passed. And you know what? Our Morning Consult partnership has been integral and important, getting that up-to-date information, and then being able, when you click on it, you can click the state result page, and you can get a PDF that breaks down the information for you. So, yeah. It does matter. It’s just important for it to be in a digestible form.
Mike McShane: So, I want to ask the same question to all three of you. Drew I’ll have you answer it first, then Marty, then Jordan. If you had your classic kind of elevator pitch, you only had 30 seconds with someone and you wanted to say the one most important thing, as you all look through The 123s of School Choice, what is the one thing you hope people take away from? And obviously it’s all great stuff, 169 studies, lots of interesting things in there, but if you only have sort of one take from it, Drew, maybe go first. If there’s one thing that people should know about what The 123 of School Choice tells us, what would it be?
Drew Catt: It just shows that a bunch of bright researchers from multiple disciplines have tried to tackle this problem to see if school choice “works.” And the result is an overwhelming, yes, it does.
Mike McShane: Marty.
Marty Lueken: So, we have a section in The 123s that talks about some of the limitations and things that should be considered. And I think that the one takeaway that I get is that no two choice programs are designed the same. Policy design matters very much. And we see that, in some programs design certain ways, you tend to see positive effects or stronger effects.
We also see negative effects as well. Not everything is rosy. And we chop that up to policy design, part of your work Mike, speaks to that as well. So, I think that, yeah, overall the studies tend to skew positive, but it’s also important to understand why we see better results in some places and not so good results in other places.
Mike McShane: Fantastic. Jordan, your one takeaway.
Jordan Zackery: Yeah. Mike, my one takeaway is that, there’s many ways that school choice can be beneficial. It can be to the student, whether we talk about test scores. It can be for parents, when you talk about satisfaction. The big thing is that giving options to students, and giving options for parents, can lead to positive outcomes across the board. And that’s the big takeaways from The 123s. It quantifies that for people to be able to read and be able to understand.
Mike McShane: That’s great. I mean, look, I think it’s so important for people to understand. Some folks think that these questions haven’t been studied or there’s like a small amount of information out there. There is a ton, right? Lots of smart people, as Drew and Marty, Jordan, all of you I think are right over the target here, which is to say, there are answers to a lot of these questions. Now, again, even I a designated Ph.D. holder here will tell you, research isn’t everything. And every study that we include, has some sort of limitations that are part of it. That’s why we try to aggregate all of this stuff together to get the overall picture of what’s going on. But every bit of research has its flaws. And look, in research doesn’t tell us everything, that there are lots of important things that we care about, that researchers can’t exactly measure.
So, if we care about empowering families or if we care about individual liberty, freedom, petition, truth, justice, and the American way, there all of these things that can be very difficult for people to necessarily quantify. But this is to say that when people have questions about what the research says, you can go to this document and it will give you, I think, an unbiased look at the state of the research literature.
Drew Catt: Yeah. I would say the question that I’ve gotten, that’s really interesting is, well what about these lifelong outcomes? Like after college? It’s like, well, I don’t know if you really understand how old some of these programs are, we’re just now getting college graduation in terms of research. So, I think the best research that may come out about school choice, is still a good 10, 20 years away, just because we have to wait for the students that were enrolled at the beginning to get old enough.
Mike McShane: For sure. Absolutely. So, I think there’s lots of interesting research that’s coming down the pike. I know many of the researchers that we include in this have other research projects that they’re working on. And I think all of us are looking forward, but we’ll see what The 123s of School Choice holds next year. We’re constantly in discussions about, should there be new issue areas that we include in this? Are there other studies that might be included or not included? And so, all of you will have to wait with bated breath, but until then, please head to our website, www.edchoice.org, check out the new improved, and up-to-date SlideShare of The 123s of School Choice. And as always, as all of us who are on this podcast right now, it’s very easy on our website to find our contact information. Do you ever have any questions? Questions about what the research says? Questions about how the research was done? Any of that sort of stuff, please don’t ever hesitate to reach out to us.
We’re happy to help answer those questions. And I think we could obviously talk about research all day. It’s basically what all of us do. So, we’ll leave it there, but it was great. Jordan, Drew, Marty, it was great having you on there. As always, we also need to thank Jacob, our indefatigable podcast producer. Thank you so much for all of your great help and stitching this thing together, when all of us forget what we were saying and need to start over. All of you who are listening to this, we all sound perfectly coherent and wonderful. And that’s basically due to Jacob Vinson’s work. So thank you, Jacob. And until next time, this has been another edition of EdChoice Chats.