The EdChoice research team talks about their list of the 25 most significant research findings over the last 25 years.
Mike McShane: Hi, welcome back to another edition of EdChoice Chats. This is Mike McShane, director of national research at EdChoice, and we have a cracker of a podcast for you all today. We are going to be talking about 25 pieces of education research. And I know as soon as you hear that, you’re saying I’m flipping to something else. Does Serial have a new episode out? Maybe Joe Rogan is going to talk about COVID cures or something. I think you’re going to want to stick through this one. And it’s funny that we bring this up because obviously this is part of our 25th anniversary celebration. I’m joined on the line today by other members of the research team, my colleagues, Paul DiPerna, Drew Catt, Marty Leuken, and John Christoff. And we’re talking about a post that all of you can see on our blog at EdChoice, where we highlight for the past 25 years of our time at EdChoice, 25 great and important pieces of education research, school choice research.
Now, picking these things is always a difficult task. It’s so funny that we are recording this in the middle of September and just yesterday Rolling Stone… And I don’t know if anyone else saw this. I’m springing this on my colleagues right now. And I know all of you are listening to this, but you can’t see the looks on their faces when this comes out, came out with a new ranking of the top 500 songs ever. Right? And I won’t do the top 25, but the top 10 songs in history, which they’re saying, and obviously can’t be in history because, I don’t know, Schubert isn’t included, but number 10, Outcast, Hey Ya. I think it’s all right. Number nine, Fleetwood Mac, Dreams.
Number eight, Missy Elliott’s Get Your Freak On. Wasn’t expecting that one, but great tune. Number seven, the Beatles, Strawberry Fields Forever. Number six, Marvin Gayes What’s Going On. Classic tune. Number five, Nirvana, Smells Like Teen Spirit. Number four, Bob Dylans, Like A Rolling Stone. Number three, Sam Cooke, A Change Is Going To Come. Number two, Public Enemy, Fight the Power. And number one, R-E-S-P-E-C-T, Respect by Aretha Franklin. So as tempted as I am to throw it to my colleagues and discuss whether we think that is in fact the proper top 10 of songs of all time, I guess we’ll talk about education research though. I will throw it out to any of my colleagues if one of those 10, if you heard a song that you think shouldn’t be in the top 10 or you think a different song should be in the top 10, please feel free to put that as you talk about education research, but I’ve done enough yammering here.
I want to ask my colleague Drew Kat, who spearheaded this effort. Can you talk just a little bit about the process of how we arrived? I don’t actually know how Rolling Stone came to their 500 greatest hits. I will probably disagree with however they did it, but something that I won’t disagree with is how you chose to do this. So how did you come across these 25 research studies?
Drew Catt: So that’s a great question. Knowing that it was our 25th anniversary kind of had this idea of, oh, let’s do like top 25 research findings and then the question was all right, how do we do it? Like, do we want to be at all internally and be like this is what we at EdChoice think. And I’m a big picture kind of person. I want to make the tent as big as possible. So I figured, hey, let’s ask our great network of amazing academics and researchers. Let’s ask the people that actually researched this themselves, what do you think the most important things were over the last 25 years? So that’s what we did. I probably emailed a good 50 to 60 different people. This was back in early January, around National School Choice Week. So not everyone responded because I guess when you’re a professor and that’s the beginning of a new semester in the midst of COVID, you have some other things to do.
But the responses that we did get were amazing. The fact that there were multiple people that said the exact same report, that was great. So many different people said the line of research by David Figley with competitive effects, which we can get into, but really the hard work was then just narrowing it down. And John Kristof really helped with helping break things out into categories. And then there were one or two that we ourselves, especially Paul DiPerna said, we have to make sure that these are included. So yes, we did add one or two ourselves, but I would say other than that, for the most part, these were all externally sourced by the experts themselves because what I view as the most important research is one thing. But to say that, hey, the top researchers in this field, these are what they said.
Mike McShane: Sure. Now, Paul, you just celebrated recently 15 illustrious years here at EdChoice. Before this, obviously with your work at the Brookings Institution and others you’ve really had an opportunity to follow the trajectory of education choice research for some time now. So I would love to give you the sort of first stab at this. So of these 25 studies, is there one or two that really stood out to you as particularly meaningful?
Paul DiPerna: I know. Time flies when you’re having fun and it has been a great ride here at EdChoice. I mean, I think some of this ties into the personal and just kind of like how I got interested in choice research and on the topics that we all study and lean on so many others externally than what they do. And so there are a couple that come to mind. And so one is the randomized control trial experimental research that was conducted by Paul Peterson and his co-authors that included William How, Patrick Wolf, and others over time. But those were experimental studies that were conducted in New York City, Dayton, Ohio, Washington, DC in the mid to late nineties. And then they really started to publish in earnest in the late 90s and then in early 2000s and culminating in the book that Paul Peterson and William How co-authored, the Education Gap: Vouchers In Urban Schools that was published by Brookings.
And so that published in 2002. And then there were, I believe at least one or two more editions that followed that, but that I think really to me, and actually just on the personal side, I was able to see that released and see Paul Peterson and William How talk about the research and including others who were discussing at Brookings, but when they published the book. And it just really seemed to be so rigorous methodologically and the investment of time and effort and the teamwork that really went into those kinds of studies. And that continued on later on with Pat Wolf’s evaluation of the DC school voucher program. It also just set the table for further research, experimental research down the road, and it showed those positive impacts, at least for some groups of students that these privately funded voucher programs in DC, New York, and Dayton, what they could do. And then that would lead to those experiment research of publicly funded programs too, like there at DC.
So that one really stands out to me. It left a really strong impression on me and also Jay Green’s work that he published in the journal, Urban Society too on Milwaukee. And those were really instrumental like in my kind of understanding of how school choice research was going and like the innovations that were happening in terms of the design and methods of research.
Mike McShane: Yeah, no, I think that’s right. And it’s interesting to see how even other fields of study within political science and economics. I mean, how influential so much of that school choice research was done and using randomized control trials and others where both in education and outside of it, there was so much work done before then that had the very selection bias issues that we would be worried about with something like a school voucher program, but didn’t use as rigorous methods. So you’re right. I mean, I think all of that stuff was so important, both for understanding school choice, understanding education policy and public policy in general. So Marty, fellow Arkansas Razorback graduate of the PhD education policy PhD program there, I would love to know what your picks are, what study stood out to you?
Marty Leuken: Yeah, well, first, I can come up with my own top 500 of the greatest songs of all time, Mike, but I’ll leave that for another podcast. But to me, one study that really stands out is Cecilia Rouse’s 1998 paper of the Milwaukee Parental Choice Program. That’s just a historically important paper, I think, as it’s the first paper published to employ random assignment and it’s research designed to study a school choice program. And really the first rigorous study of the oldest modern day voucher program in the country as well. This was published at a time when that program was expanding to allow religious schools to participate. And I’m not sure like what role, if any, Rouse’s study had in that particular expansion, but the program certainly had a lot of room to grow and to improve. I got to agree. And Wisconsin needed to demonstrate the effectiveness of the program in the years to come to help it evolve in a positive direction. And I think Rouse’s study certainly played an important role in that.
Mike McShane: No, I think that’s right. And it’s so interesting that she publishes that study in 1998 and has gone on to a series of incredibly prestigious places. Marty, am I right? Is she the president of the Council of Economic Advisors for President Biden? She’s on the Council of Economic Advisors or is she in charge of the whole shebang?
Marty Leuken: I believe she is in charge of it.
Mike McShane: Yes. And that’s being backed up by nods in other places. Yeah. So that’s great to see someone who did such a seminal piece of work in this field go on to such important and lofty post. Drew, I would love to know your pick. So we’ve got some randomized control trial fanatics here. What did you see?
Drew Catt: Honestly, for me and a lot of this is coming from my background and philanthropic studies and working for a… Having education and youth development under my grantee portfolio and what seems like a previous life from now. But the thing that really funders cared about was graduation and going to college. Could schools do that? If so, yeah. We’ll throw them more money. Can programs do that? If so, yeah. Okay. Let’s throw them more money. So what about school choice?
John Kristof: For me, it was Matt Chingos and Paul Peterson really looking at school vouchers on college enrollment and degree attainment. Really, it was starting to look at New York City, which is a privately funded program, but Matt has gone on to do the same thing, looking at Florida, looking at Milwaukee. I mean, the results are the same and same again. Low and behold, do test scores matter? Questionable, Ben Scafidi has a nice survey piece on that about ‘Do they matter to parents?’ which is also included in the list.
Sorry, if I stole that from anyone else that was going to reply with that one. That’s also influenced my parent survey work, but really it’s showing that, Hey, these programs actually impact students and the levels at which they graduate from high school and the levels of which they go on to college and depending on whether it’s full sample or some subgroups, impact how soon they get their bachelor’s degree or whether or not they receive a bachelor’s degree. And I think at the end of the day, most people can agree that when you look at salary metrics or anything, the differentiation is whether or not you have a diploma. So I think the fact that Matt and Paul were able to do this work and to see it replicated in other places is fantastic and is amazing for the movement.
Mike McShane: Agreed. So John, you are perhaps the most recent addition to our Merry band here. And so I’d be interested to sort of… You kind of came into this out of a different kind of generation of researchers. Obviously, we came from our particular backgrounds and training. So I would love to know you as someone who’s come at this a bit more recently, what stood out to you? What was meaningful for you?
John Kristof: Yeah. Drew mentioned this a little bit earlier, but I’m going to throw a shout out to David Figlio and company and his kind of initial line of research on the competitive effects of private school choice. My background in academic thought is in economics. So I have my background with knowing Milton Friedman and his impact on economics and education policy and economic history, plenty of things. And so, market design has always been very interesting to me and what school choice argues and what Milton Friedman argued when he was promoting this idea of school vouchers decades ago, was that the education market, which certainly at his time, no one was thinking of education as a market will be more efficient if you allow students to move different places, to go to attend different schools, including private schools that would charge tuition. And if the market theory is correct, that competitive pressures will encourage schools to… Whether it’s be more efficient with their resources, be more intentional about curriculum, et cetera.
Then we should see positive effects, not only for students who are able to take a school voucher or some other form of private school choice and apply it to a private school, but we also should see benefits for public school students who don’t use vouchers for whatever reason, because in theory, the public schools should be responding to this idea of competition. And Figlio and Hart’s 2014 competitive effects study of the Florida tax credit scholarship program found this fairly convincingly in a couple ways. First, they found that the effects of competition were immediate and at least part of the effect had nothing to do with selection of students moving from one school to another, because there was a delay in when students could move schools, actually use the vouchers. So the very year that the Florida tax credit scholarship program passed, again, no students had used any scholarships yet.
There was a 0.01 to 0.02 of standard deviation improvement that was statistically significant in public school student test scores. So, that’s not world changing, but after one year there is… That would translate into a month or two of extra learning that a public school student got. Again, without the private school choice program even happening, just the very presence of competition and the promise of future competition seemed to spur public schools to serve their students a little better or respond, in a way. And the second thing that Figlio and Hart found was that the higher the degree of competition a public school faced, the stronger these competitive effects were. So Figlio and Hart used five different measures of private school competition because, I’ll speak from personal experience, it’s very difficult to measure in a lot of ways.
So you often see researchers in this field using a lot of different measures to get the best guess of competition that they can. And in short, basically, for example, the closer a private school was to a public school, you might see that as it is more likely that parents would see that as a viable alternative, because transportation is less of a concern. So considering a measure like that, the more competition, the closer and the more direct competition a private school provided, the stronger the competitive effects were. In other words, the more threatening a private school was to attract students from a public school, the better public school students did.
So in this initial study, and this has been confirmed through other studies from Dr. Figlio and it started a line of private school choice research in this area that I know Drew contributed to suggests that this idea of market competition really does improve the education experience for students everywhere, whether or not they choose to or are able to take advantage of vouchers. It just improves all options, even more so. And these effects roll over and continue year after year, which is another finding of his line of research. So, big fan of Figlio’s work. I’m happy to see it included in this report.
Mike McShane: Yeah. So I guess I’m going to bat cleanup here. I will add just continuing on the Rolling Stone theme, I probably have my quibbles with the top 10, but the single most objectionable thing that I see in that list at number 19… Supposedly 19th best song ever is John Lennon’s Imagine. John Lennon’s Imagine is sandwiched between at number 20 is, I believe, Robin’s Dancing on my Own, a dance kind of disco classic. And number 18 is Prince’s Purple Rain, which could have easily been a top 10 hit. And you have John Lennon’s Imagine, which shouldn’t even be in the top 500, because it’s one of the worst songs that was ever written. It is a terrible song and if I never heard it ever again, it would be a blessing. But yeah, so that’s my-
John Kristof: At least I know that our list is not going to be the most controversial list to be released this time of year.
Mike McShane: Thank you. That’s exactly right. So we’re just going to be in the slip stream of Rolling Stone as they make such an objectionable decision is that one there. But anyway, I will, since you all named so many great studies, the one that I will highlight and it’s as much for one of the people that was involved in it and just sort of a personal connection, but Pat Wolf at all’s study of the DC voucher program. I have a particular sort of connection to this because I was Pat Wolf’s research assistant when I was in graduate school and Pat Wolf has done some of the most important research on private school choice around the country. So he was parts of the teams in Milwaukee, in Washington, DC, eventually in Louisiana, and so all of these things that he really pushed doing randomized control trials, he worked with the federal government, he worked with IES and others.
So really went through all of these hurdles to, I think in many ways to really legitimize school choice research and dotted every I, crossed every T, did everything that he needed and with the sort of imprimatur of the federal government and others. So I think obviously that was a really important the DC evaluation that was done by the Institute for Education Studies and had to go through all of these. And I just remember I played the tiniest of tiny roles in that, but I sat in on a whole lot of conference calls of all of the steps that had to be taken and all of the review and everything that had to be done for the federal government to release a report. And so, having someone who was willing to take that on and took the science so seriously and thought it was so important what he and his team were doing, I just really value that.
And I think Pat Wolf really deserves a shout out because I think… And it’s demonstrated by the results of the studies that he’s published. He’s just one of the most honest, forthright researchers that we have in this space. So when he found results in places that were positive, that’s the results that he reported. When he found results that were neutral, those are the results that he reported. And when, in places, he found results that were negative, those are the results that he reported. And he really subordinated his own particular views to the truth, and it was really something that he instilled in a lot of us when we were his graduate students, that he said, “One of the best ways you can make the world a better place is to just tell the truth,” because that’s what we’re here to do as researchers.
So I think that it’s one of the reasons we can trust the research that’s included in the list that he’s done, because he’s obviously shown himself to be such a person of such incredible integrity. And I think is really a role model for all of us that are involved in research, whether we’re at EdChoice or elsewhere. So friends… Yeah, go ahead, Paul.
Paul DiPerna: Yeah. I 100% agree. I think, yeah, I mean, we owe a lot to Pat and the work he’s done and he’s really blazed the trail on school choice research for so long. The only thing I was going to add and this really can connect you and to Marty as well, but it’s interesting to see on that list how many of the students who have come out of Arkansas and from the department of education reform there who have gone on to publish in their own right, whether they’re reviews of the research like Danny Shakeel has done so well with, Caitlin Anderson, and Pat very recently. There’s a global study, looking at private school choice in a number of different countries, including the US. But yeah, I guess for lack of a better reference, but there’s a coaching tree.
Mike McShane: I was just going to bring that up.
Paul DiPerna: Very impressive that I think Pat has. And also of course, with Paul Peterson.
Mike McShane: That was connection I was going to make was, if you really think about it, this is all a big Paul Peterson coaching tree, where you had… I mean, it would be interesting and I don’t know if we did this, but in this list of 25 people of all of those things that are somehow connected back to Paul, because it was, I mean, I just know so many of us. Yes, and I think Drew brought up as well and David Figlio as well. I mean, so there’s a couple of these… Yeah, borrowing from the NFL, the coaching trees, but many of these roads lead back to Paul Peterson and Pat G. at Harvard.
Paul DiPerna: Yep. Yes. Yeah. It’s really interesting just to see that and it’s a real true testament to them as not only scholars who are very impressive in their own right, but also as mentors and building and supporting students in all the right ways, and to help them publish and advance in their careers and lives.
Mike McShane: Okay. So we’ve talked about our favorites, but were there any others, maybe other ones on the list that stood out to you or maybe some honorable mentions that just missed the list? But Paul, were there any other ones that stood out to you?
Paul DiPerna: So yeah, there are a couple. So Terry Moe’s book, a public opinion book, that really was focused on vouchers and this was published back in 2000, but it still… I think it was hugely influential on me just thinking about public opinion around school choice and K-12. So I just want to make sure that we mentioned his book, Schools, Vouchers, and the American Public.
Mike McShane: Cool. I’ll definitely shout out the folks that really work to try and do either meta-analyses or combinations of things. There’s so much work that’s been done on that. So the folks who did really careful work, there’s a couple on the list there, but I certainly appreciate trying to make sense of all of the work that’s out there.
Marty Leuken: And I’d like to point to Susan Aud’s fiscal study for the, at the time, the Friedman Foundation. That was the first national scope fiscal study done, which spawned a lot of other fiscal analyses of which I am part of and doing right now. But that was really the first one, which to address what we all, I think, agree is the most pernicious argument against choice, which is drains resources and harms public schools.
Mike McShane: Yeah, I think that’s a great point, Marty. It’s something we probably should have said already that I know at least I feel as a school choice researcher, and I know you specifically with all the great work you do on education finance, but really is a situation where we stand on the shoulders of giants. We are able to do the work that we do because of methodological advances that folks before us made, questions that they asked and answered, that we were able to learn from and that we’re trying to build on what they had to do. But that’s a really good point. I’m glad you highlighted Susan’s paper because that really is a foundational thing that so many folks have built off of. Drew, and then I think John.
Drew Catt: Yeah, real quick. This is one that I mentioned, Ben Scafidi’s More Than Scores, really looking at asking the parents of the students participating in these programs, “Hey, why’d you choose the school that you did? Why’d you choose this program?” And then Paul building onto that and doing more of the participant versus private school comparison, which then led to me opening up the can of worms and saying, “Hey, let’s just ask all types of school families and let’s see what the comparative advantages are and if there are similarities or differences, regardless if the parents are choosing traditional public, charter, private, or through a program homeschool, whatnot.”
John Kristof: Yeah. Drew stole mine a little bit, but I’ll add on why I think it’s important to me and that’s that the research community and oftentimes people more broadly, they ask, “Does private school choice work?” And what they often think of is test scores. But when you look at what parents actually want, one of the lowest scores for reasons that parents in this Scafidi study, one of the least common reasons that parents exercised a voucher and chose a private school was test scores. Much more important things were things like student safety and student discipline, and just an overall better learning environment.
And if those are things that parents are actually more interested in and those are the kinds of things that contribute to much higher parental satisfaction rates, then we also have to consider that when we’re asking these big philosophical questions of, “Does school choice work?” And I think that’s something that we, as researchers who obsess over numbers and things like that can often be reminded of and be helped by that.
Mike McShane: For sure. Well look, gentlemen, this has been a pleasure. Oh, Drew, you wanted to say something? Since you put all of this together, I’ll give you the last word.
Drew Catt: No, it’s just fascinating. So we have the Goodnight, whatever Moon. There’s a series. Since my wife is a chemistry teacher, somebody gifted us Goodnight Lab. And one of the things is goodnight professor that’s saying publish. And my niece was reading my three year old this book the other day and my instant reply was, “Yeah, publish or die.” And he’s like, “What does publish mean?” So it was really going into like, “Okay, what does this actually mean? Why do people do this?”
And just the fact that there are people out there who, regardless of whether or not it’s part of their job requirement, they want to do high quality research and they want to study these things and get it out there. So I’m just so appreciative of everyone that one, wants to study school choice. And two, wants to do it in a rigorous enough way that it gets placed in top publications or is put out by amazing organizations. And yeah, just getting it out into the world and increasing the awareness that not only what school choice is, but hey, school choice works.
Mike McShane: Well said, Drew. And you know what? Y’all, here’s for another 25 years. Here’s for another 25 years of EdChoice. Here’s for another 25 years of interesting education research, interesting education choice research. There are so many bright people trying to tackle a new generation of questions that are out there and I know we are all excited to continue to learn from them. As always, please like and subscribe to this podcast. Drew has a great series where he interviews interesting education researchers. And so if you’re interested in the folks that are in this field, that’s part of this podcast series. Our colleague, Jason Bedrick does this Big Ideas series where some of these folks are specifically school choice researchers and other people are just interesting folks in the education space. So I think both of those are a reason to subscribe to the podcast because you can hear more about that.
You can also hear we talk every month about our monthly polling that comes out. We talk about our Schooling in America survey every year when it comes out. So lots of great stuff. So please like and subscribe to this podcast. As always, I want to give a big shout out to Jacob Vinson who edits these things and makes us sound more coherent than we actually are. And I appreciate all of you for joining us and listening to us talk about research for half an hour. And if you’re still listening at this point, which I imagine that you are, I hope you learned something. We really enjoyed talking about this and I look forward to talking to you again on another addition of EdChoice Chats.