Learn more about our recent brief, Comparing Ed Reforms. We partnered with Hanover Research to assess the experimental research on nine K-12 education reforms.
Drew Catt: Hello, and welcome to another episode of EdChoice Chats. I’m Drew Catt, EdChoice’s director of state research and special projects. And I’m here to talk about our newest research, a brief called Comparing Ed Reforms, assessing the experimental research on nine K-12 education reforms. I’m here today with Robert Enlow, EdChoice’s president and CEO, and Paul DiPerna, EdChoice’s vice president of research and innovation, to have a conversation about the research. Thanks for joining me today, Robert and Paul.
Robert Enlow: Thanks for having me.
Paul DiPerna: Hey, thanks, it’s great to be here.
Drew Catt: So Robert, would you mind starting us off by talking about why we did this research and what inspired it?
Robert Enlow: So, great question, Drew. I’ve been involved in EdChoice now, or Friedman Foundation or private school choice for 25 years and it seems like every other year, some new type of reform comes out, which is great. It’s important that we think about how to improve quality for kids. But whether it’s school takeovers or class size or pre-kindergarten or open enrollment, all these things have come about in the last 25 or so years. And so as charters and private school choice have been around for that period of time, I wanted to find out and I wanted our team to find out what the heck is going on with the number of gold standard studies comparing these reforms? What do we know about the number of studies compared to the charters and choice and what do we learn from those? So, it was important, I thought, for policymakers and advocates just to have a real clear understanding of what the heck the numbers said about the numbers of studies being done and what those studies that are high quality actually say about the positive or negative effects.
The reason why is simply, I think, it’s important for everyone to know what kind of studies have been done and what are the results of those studies. And policymakers and advocates need to actually have a grip on those numbers because it’s important not just to run to the shiniest object that we can see, and start looking at these things over time with a high quality survey. So, that’s sort of the reason why we at EdChoice did it. I’m super happy and proud of our team for getting it all together.
Drew Catt: Yeah. I can kind of relate to chasing after that bright and shiny thing as someone who describes themselves as a magpie that flies to those things. It’s great to have someone that is looking at that overall time horizon to kind of see what is and is not working. So Robert, you mentioned a few different audiences there. Who would you really say that this research is for?
Robert Enlow: Yeah, it’s a great question. To me, it’s really for two groups, right? It’s for advocates of all types—people who are reformers on the right and left. Just so that they know the number of gold standard studies because as you know at EdChoice, we care deeply about gold standard studies and not just any studies. And so advocates are the number one audience. The next audience that’s most important is policymakers. And these guys and women are getting bombarded every year in the legislature with, look, this is a really cool idea, let’s do this. If you just give me the money to do innovation districts, or if you just give me the power to do portfolio management or site based budgeting or all those kinds of shiny object things, then we’ll be able to understand what we need to do. They get all these pushes from everyone outside and they need to know, and policymakers need absolutely to know what the studies say. So, it’s policy makers and advocates
Paul DiPerna: Just to follow up with what Robert just said, I totally echo that, the policymakers can really use this research and reviews. And our partners at Hanover Research, great partners of ours that we’ve been working with for a number of years now, they put a lot of time and effort into this. And I think this saves time for policymakers where we know that even before the current pandemic crisis where time is just so precious and attention is so precious, especially for them, even before that, they just don’t have a whole lot of time to do the due diligence and the kind of landscape research to see what’s out there, especially on the RCTs and experimental side of research. And so we hope to provide this as a service to policymakers. I also want to add one more group in addition to policymakers and advocates, it really should be a useful primer to new education researchers and students, truthfully. And so this really should reach a really broad audience there and a usefulness there.
And going back to that theme of saving time, we have so many sources cited and reviewed in our main EdChoice brief, but then we also have a couple of supplemental resources as well that Hanover put together for us. And so that should really be useful to researchers, especially new ones, and students. And so we hope that there’s some broad usefulness to what we’ve done.
Robert Enlow: That’s a great point Paul—
Drew Catt: Yeah.
Robert Enlow: So, we can add a third category, researchers, sorry. So, policymakers, advocates, and research so that everyone could be on par.
Drew Catt: I’m going to throw a curve ball in here and add another category. And to be honest, this is the kind of thing that I would have loved to get my hands on when I first started out in my last job in education grant making. It’s kind of told that any good reform takes at least five years to really see any reliable results. And to be able to have something like this looking at what is and is not working, it really would have informed the foundation’s grant making. So, Robert used gold standard, Paul you talked about RCTs. Would you mind to tell our listeners a little bit about what gold standard and RCT really means and kind of dive into the methods for this research and what people can find in the report?
Paul DiPerna: Sure. Yeah. Happy to do that. So yeah, we use those terms interchangeably, whether it’s experimental research, randomized control trial—or otherwise known as RCTs types of research—and then also we use gold standard. So, those three terms we’ve used interchangeably, not just in this brief, but just the way we present and review research and our other publications as well. And so what we mean by that is that there is a treatment and a control group comparison for these studies that we reviewed and that Hanover reviewed for us. So, there will be this important comparison of those who are either exposed or use the intervention, whether that’s private school choice voucher programs or charter schools, or participate in reduce class sizes, pre-kindergarten interventions. And so that comparison’s important. For it to be as rigorous as it is we see that you control for selection bias and self-selection.
And so that’s something that makes this type of research more rigorous and what many researchers in the scientific community feel to be superior to quasi experimental or non-experimental studies. And so this types of research has what we call a high degree of internal validity, where the effects that we observe, we can have a high level of confidence that, one, the methods are reliable and that we can believe and be confident in those findings for those effects on that population by way of that intervention or treatment, if that makes sense. And so we looked only at this type of research, the randomized control trial experimental research, we didn’t look at quasi-experimental or non-experimental studies. And then we focused on academic outcomes or education attainment outcomes. And so the academic outcomes could be like test scores and of course completion or for the pre-kindergarten the academic skills that they acquire through pre-kindergarten. And then attainment is high school completion and graduation, college entrance and college matriculation or college completion.
So, those are the types of outcomes that we focused on. And that was another inclusion criteria that we had included in the research design that we pursued with Hanover. So they should focus on those two outcomes, they should be randomized control trials or experimental research designs. And then we also looked at the populations that were observed and studied, and if there were more than one experimental studies conducted on the same sample of students that was only counted as one study, not multiple studies. Especially when it comes to the class size research and the STAR project in Tennessee, that’s an important thing to note because there have been numerous experimental studies on same cohorts or same samples of students that participated in the STAR project in Tennessee. And so that’s another factor that we had included in our study design and review design. And then for those who are interested, we have an appendix that goes over that study inclusion criteria and the list of the databases that we used for our searches and to locate the studies.
Robert Enlow: Paul, that’s a really incredibly thorough explanation. I really appreciate that, and I know our listeners will too. As you said, we’re looking at gold standard, random assignment, experimental studies, right? Basically those are the studies that say here is the pill and here’s the placebo or here’s the treatment and non-treatment and everything else about the child or the parent or the person being studied is absolutely made similar. So, the only difference is whether they get the treatment or they don’t get the treatment, or whether they get the pill or they don’t get the pill. This is why these tests are so important. It’s because you really can say with confidence that it’s actually the treatment that is making the difference or not making the difference here. These other types of studies, like the quasi-experimental studies you’re talking about, which we call silver standard are basically saying, it’s likely that the treatment is the reason, but we’re not 100 percent confident, we’re not even 90 percent confident.
And then there’s the bronze standard one which is basically telling you, well, we liked the treatment, we saw it on Tuesday, we don’t know if it has any impact, right? It’s these observational studies. And this is why we focus at EdChoice so much on the gold standard, random assignment, experimental studies. So, thanks for that explanation.
Paul DiPerna: Yeah. I think that pill placebo example, that’s definitely useful to talk about this. Just to note, there are other types of outcomes that we’re not able to do randomized control trial studies and that’s where the quasi experimental types of designs are very helpful. But for this, we really wanted to focus on the experimental research because I’m thinking, Robert, you know this way better than I do and all the conversations you have with legislators and policymakers, I think that’s the first go to. What they’re interested in is what does the most rigorous research say? And so that’s why we wanted to focus on the RCTs and gold standard research.
Robert Enlow: To be honest, we want to focus them on the most rigorous research so they don’t get easily drawn into the bright, shiny objects. Look, Drew, we can start getting to this now. I was blown away. If you’re looking at the results of our Comparing Ed Reforms, there’s a couple of things that blew me away. One, the vast number of random assignment, gold standard studies that have been done on charter schools and private schools compared to any other type of reform we did. So, the next nearest reform that had the highest amount of random assignment studies was prekindergarten and that was eight. That tells me that private school choice and charters had been studied with random assignment studies two and a half times more than the next highest ed reform. Heck, some of the newest stuff about portfolio management and common application school takeovers has zero random assignment studies, and school size has only three. Legislators and advocates who are making pronouncements about how important these reforms are should really look at the rigorous studies before they say, “Hey, we love this idea.”
This should be about, as Drew said, giving a reform over time to see if it’s working or not. So, that’s one of the things I think is really important about this. This isn’t really trying to say one reform is necessarily better than the other, but we do want to say the research is actually the research and here’s, what’s been done, and here’s what we know about it. And so let’s not play games with the numbers. The numbers are that charter schools have 22 studies, and private school choice has 21, and prekindergarten has eight, class size has four, school size has three, open enrollment has two and common application portfolio in school takeovers have zero. These are really important things to know for policymakers and that’s what I was blown away with, the amount of studies that have been done into the private school choice and charters versus all of these others.
Paul DiPerna: Yeah. Robert, I totally agree. That is something that I, for one, didn’t realize that charter school and gold standard studies have caught up and even surpassed the number of private school choice studies. And that is the main takeaway here, it’s just the volume of research, the expansive body of research right now on charter schools and voucher programs compared to those other types of reforms. And I think that’s something that has been overlooked and hopefully this can bring some more attention to that. Just how much more gold standards research has been conducted on those two reforms. And we’re not really pitting one against the other, that’s not the main point. It’s just that, hey, look for those who don’t think voucher programs or charter schools have been studied or studied enough, this should be an eye opener.
Drew Catt: Speaking of not pitting things against each other, I’m going to be honest, those pre-K results showing that a quarter of the studies are showing negative effects is a bit of a hot button issue around my household. We have an almost two year old, and my wife and I have begun the conversation of whether or not he should go to pre-K and what kind of pre-K and I’m kind of leaning towards where the research is going that well, it may not be the best situation for him and we’ll just gather more information and see what’s best for our individual student.
Robert Enlow: Hey, that makes all the sense in the world, Drew. And if you look at even the positive studies on the pre-K, there are people who are saying that those studies are coming from a group of like 30 to 60 students, and there’s just no way to massively replicate the kind of money or services spent to get those kinds of results. And so I think you’re right on by saying, let’s take a look at what your individual child needs and find the right environment, whether that’s a pre-K program or continuing to be at home.
Paul DiPerna: Yeah. And along these lines, this is something for those listening who haven’t had a chance to take a peek at the brief yet. Looking at the locations too, and seeing where these studies have taken place and you’ll see like on the charter school map there are many locations where there have been RCT studies conducted. And then you see fewer places and some other areas like the pre-K or class size where it’s one State. And so something else for readers to consider is that, what might work in one area may not be as effective, or we just can’t draw conclusions outside of where that study was situated.
Robert Enlow: Exactly. Right.
Drew Catt: So, Robert, do you have any recommendations for the folks who can actually affect policies in these areas and in states?
Robert Enlow: Well, here’s a crazy recommendation, pass policies in line with random assignment studies. The greater the number of random assignment studies, the greater likelihood you pass a policy. That’s a simple tradeoff for me. If you know what the results are going to be and you’re confident in them, you should pass those kinds of policies versus not. If you don’t know through random assignment studies what the policies tell you the results are, you should wait and maybe find out until the data comes in.
And I think this is really important, mostly for educators because they’re tired of having to be played around with and experimented with. And they’re being experimented with all these new types of reforms. If we know that the reforms based on parent choice are working better, pass those. Those by the way are the ones that if you look at states like Indiana, when you pass all the reforms, the whole reform package was passed in 2011. The only thing left standing is our charter and voucher program that’s been unaffected. In fact, those have expanded since then, everything else has been sort of not expanded. And that’s because it’s all based on a system as opposed to based on a parent. And so my number one recommendation, Drew, as I sign off here is policymakers should pass a program based on the number of research studies that have been done.
Drew Catt: I think that was a great way of summing that up, Robert.
Robert Enlow: Yep.
Paul DiPerna: I think policymakers but also other folks in our orbit of ed reform or those who want change in the system… Just attention, time and attention, they should pay attention more to these reforms, myself included, where there has been more research and where the more rigorous research has… So, Robert covered the policy side and the policymakers, but I think among like advocates, activists, reformers, we should be paying attention to these areas that have been studied more rigorously over time.
Drew Catt: Yeah. So, Paul, are there any plans to do any more research in this area or at least keep the information current as time passes?
Paul DiPerna: It’s a good question. Not right now. This was something that, maybe you should have said this at the beginning of the podcast, almost a decade ago, Russ Whitehurst at Brookings had a really interesting, and to me, just impactful piece looking at the experimental research effects on various types of interventions, and looking at a different set than what we looked at. And to be honest, went deeper in terms of analyzing those types of reforms and the size of those effects for those reforms. So, that was something that Robert and I had been talking about for some time, and it would be interesting to look at these different reform areas. And I think ideally we’d love to revisit this say in three, four years from now, can’t see this being an every year type of thing just because the pace of research doesn’t really warrant that either, especially experimental research. You might only find one or several new studies every year or 18 months. But I could see us revisiting this in the longer term, say three, four years out and just see what new research is on board for these different reform areas.
Drew Catt: Yeah. It’ll definitely help that we’re continuously tracking what is happening within the private educational choice research. In fact, here’s a short plug, our 2020 edition of the 123s of School Choice is coming soon, make sure to get on our website to order your physical copies when you can. So, Paul, do you think the existence of lotteries within charter schools will make it so that there may potentially continue to be a drastic increase in RCT research for charters versus private school choice since we’re seeing not as many new private school choice programs with lottery mechanisms?
Paul DiPerna: Yeah. That’s a great question. It’s an important question because some of these, broadly speaking, is just the nature of the policy design of charter schools lends itself a little more to this type of experimental research and more so I would say than these other reform areas. And so I would expect in coming years that we will see more charter school RCTs come online more quickly than probably the rest of these reform areas. In private school choice world we do see evaluations being conducted like by Pat Wolf and his team in D.C. and Louisiana. I would assume that we will continue to see some RCTs there, but the pace is incremental in terms of RCTs coming online. To be honest, just my background, I don’t know enough about where things are moving or what’s planned and those other reform areas, but I suspect that charter schools will probably accelerate in terms of their gold standard research more quickly than other areas. And that’s in large part to the design and the use of lotteries with charter school and that lotteries are not as closely to design and those other programs.
Drew Catt: Yeah. And for those of you that are more visual learners such as myself, if you’re really interested in learning more about what gold standard research is and what an RCT is, we have a wonderful video that is on our YouTube channel. So, feel free to check that out. That’ll also be hitting our social media channels when we’re doing our push of the release of the 123s. So, Paul, before we go, were there any things that really surprised you or things that you weren’t expecting to find or anything else you’d like to highlight from Comparing Ed Reforms?
Paul DiPerna: Yeah, I think we touched on the big ones. The number of charter school experimental studies, that did surprise me, I didn’t know it was quite up that high. And I think just my better understanding of where the experimental research is in the other areas and just the limitations in certain reform areas that really prevent experimental research like for portfolio models or even open enrollment and the common application unified enrollment systems that we’ve seen come up in the last 10, 15 years. It may be even more difficult to do RCTs in those areas. Then I really appreciated the depth that Hanover went and their reviews, the courage to reinforce that and shed light on that.
Paul DiPerna: And I just encourage, just to wrap things up on my end, the listeners, to the extent that they’re interested and especially for those students out there, Hanover did provide some really nice supplemental files for us that go a little bit deeper, they’re almost serving as appendices to the brief that go deeper with more summaries on the findings and the designs of these different types of studies in all these different reform areas.
Drew Catt: Yeah, that’s great. I love that we provide all that information. Here at EdChoice we love feedback. So, if you’re listening to this, please take a chance to check out Comparing Ed Reforms and let us know what you think. Positive, negative, neutral, what have you, and then give yourself a pat on the back for having the desire and drive to learn more. To stay updated on the latest school choice research, legislative news, and more, please remember to subscribe to our EdChoice Chats podcast, wherever you get your podcasts, for more of our coverage of new school choice research, education reform policy chats and more. If social media is more your thing, follow us on Twitter, Instagram and Facebook. You can find us @edchoice. Thanks again for listening. And until next time, take care.