Ep. 338: Researcher Profile — with John Garen

August 18, 2022

In this researcher profile, we meet John Garen who is the author of a preliminary report titled “School Choice and Competition. Has there been enough enabling legislation to generate broad-based effects?” John discusses some of his findings from the research.

Drew Catt: Hello and welcome to another episode of EdChoice Chats. I’m Drew Catt, EdChoice’s Director of State Research and Special Projects, and I’m here with my colleague, John Kristof, EdChoice’s Research Analyst, to talk about some interesting research, a currently preliminary report called “School Choice and Competition. Has there been enough enabling legislation to generate broad-based effects?” We’re here today with the author at John Garen is the BB&T Professor Emeritus of Economics in the Gatton College of Business in Economics at the University of Kentucky and the Founding Director and an affiliate of the Institute for the Study of Free Enterprise to have a conversation about his research. Thanks for joining us today, John.

John Garen: I’m very happy to be here. I think EdChoice is a great organization and happy to chat with you guys.

John Kristof: Yeah. Thanks for chatting with us, John. To start off the conversation, could you introduce yourself to our audience a little bit, tell the listeners just a little bit about yourself, whatever you find kind of interesting to this discussion, and I guess particularly how you got involved or interested in researching school choice?

John Garen: Sure. Well, I’ve been a faculty member at the University of Kentucky for 36 years, just retired in January, and I’ve been interested ever since I started studying economics back in the 1970s, which is a long time ago now I guess, always very interested in how the market works and how it drives not only sort of individual market outcomes, but also says a lot about how society works and historical events play out and whatnot.

So I got my Ph.D. in economics back in the early 80s and started in academic jobs and did a lot of work on various topics, but I was always interested in school choice for a long time and didn’t do a lot of heavy-duty research on it until the last 10 or 15 years. But it’s been a topic that, as I say, I’ve been interested in for quite some time, and it sort of ties together things that a set of my interests. One is sort of human resources and labor economics, as well as understanding public/private differences in outcomes for organizations being private sector versus public sector. And as you guys know, much of the education sector is driven by public and government. And I think examining how private education and support of that can operate is very, very important issues to look at. So that’s driven my interest in the last 10, 12 years or so on this.

Drew Catt: Wow, that’s great. Yeah. We could easily have a very tangential discussion over whether private schools are actually private, some majority of them have nonprofit status, but that’s a topic for another day. But let’s transition to the research that John and I saw you present at the Midwest Institute’s Conference the other month. So would you mind telling us why you conducted this particular research and really kind of what inspired it?

John Garen: Well, I think there’s been a lot of research on looking at individual students and seeing what happens when they get into a private school through a voucher or some other assistance program and comparing them to students who are not able to go, as well as with similar kinds of research for charter schools, where kids who get in versus kids who don’t, and the results, generally speaking, have very strong effects, very strong, positive effects on the kids who get into the private schools or the public schools.

But what about, I guess the issue is often arisen, what about, is there some kind of a competitive effect on the public schools that kind of compete with these private options? And there’s been some studies on that as well and they tend to find that schools near the private options or the charter schools, regular public schools, tend to do somewhat better.

The question that I’m looking at in this research is what about really broad-based effects? Like I’m looking at the effects on statewide NAEP test scores, and your listeners probably know NAEP, the national assessment of educational progress, test scores, sometimes called the nation’s report card, national tests that people use to gauge what’s happening out there in a lot of the K through 12 education.

Anyway. So what happens to statewide test scores when states adopt choice types of programs? And honestly, I didn’t know what I was going to get walking into this because a lot of choice programs are really very limited. And I know the EdChoice folks, you guys have been doing really good research on this, and school choice programs sometimes really don’t give you a heck of a lot of choice, unfortunately, they can be really limited in the number of people who are eligible, the funding that follows the student, and restrictions on what parents can do with the funding, restrictions on what schools can do if they’ve received the funding and so on. And similar comments apply to charter schools as well.

So the question is can these have an effect? Has there been enough enabling legislation to generate broad-based effects? Have these programs been enough to really do anything broad-based? And for the reasons I just described, I had some skepticism about whether it really could move the needle on a whole state’s test scores. So I gathered a lot of data, a lot of it off the EdChoice website on school test scores from the early 90s all the way through 2019, pre-COVID, and on school spending, and on EdChoice’s tracking of school choice legislation state by state, as well as charter school data, whether the state has charter school legislation and how enabling that legislation is or how restrictive versus enabling, and put the data together and started doing some statistical analysis.

And I was quite surprised on how strong some of the effects were for choice programs, particularly states that have adopted a voucher program or an education savings account do remarkably better than states who have not. And we’re looking at how much a state improves itself relative to other states. So, that’s kind of the statistical method. How much did state X improve when it adopted a program versus how much did state Y improve that did not adopt a program? So there’s also quite remarkable.

And I was at first a little taken aback how sizable they were. And I talk about that in a minute. So I did a whole bunch of what we call sensitivity analysis to see if maybe there’s a couple of states that is really kind of a strange outlier that was driving the results, and it turns out that’s not the case, and looked at different time periods in the data, and the findings are really quite robust.

So in terms of the magnitude for folks that are familiar with this, I can talk on and on, but for folks who are not, it can get a little jargony here but turns out people measure the effects, oftentimes, not just on the raw test scores, but as a percentage of a standard deviation in the test scores, and a half a standard deviation change in a test score is really quite large.

And what I found is, well, first, I’d also compared this to the effects of school spending. And what I found is, on school spending, I found effects similar to what other literature finds, positive effects, but very, very small, like 3% of a standard deviation. But for states that adopt a voucher program or education savings account, the effects are like 10 times as large on test scores. So instead of 3% of a standard deviation, you’ve got 30% of a standard deviation gain, and that’s just remarkable.

So, like I say, I was kind of stunned at the first result of this so I did all this sensitivity analysis on…I mean, there’s more details here, but a bit of a flip side to this is I also looked at the effect of these various programs on state per pupil funding. And so, as it turns out, perhaps not surprisingly, states that adopt in particular ESAs and voucher programs actually spend less per pupil than other states, and it’s a pretty significant amount. So it’s almost equally remarkable where states that adopt, especially vouchers and ESAs, not only do their students score better on the test scores, but they’re spending less money. It was kind of a double dividend there, you’re getting more and spending less. So that’s kind of the guts of the results. I mean, there was lots of details here and there and there’s lots more stuff that I need to look at. As you mentioned, this is kind of a bit of a preliminary analysis, although I think there’s enough results there to say, “Well, there’s really something here.”

John Kristof: Yeah. That’s a very interesting, preliminary as it is, the fact that you found such significant changes in two types of private school choice reform versus others. I think it’s actually a very interesting thing to dive into. Given those kind of initial findings to maybe concretize this a little bit, what kind of implications do you think this should have on policymakers or people advocating for policy in this kind of space? What kind of takeaways do you think they should have from I’ll be a preliminary from these findings? Because I know your kind of initial question was: what has been able to pass in private school choice reform? Your question was: is it enough to actually institute these market principles into the education ecosystem? It sounds like, to at least some degree, the answer is yes, based on what you’re finding. So is that the right takeaway? If so, what should policymakers, people interested in advocating in education policy, how should they be reading these results?

John Garen: Well, I think that is the right take. Honestly, when I saw the results, both in regard to the effects on test scores, as well as the effects on spending, my eyes kind of popped. I said, “Wow, is this really correct?” And I think, largely, it is. So that’s the kind of reaction that I would hope other people would see as well as, “Okay, well, if this is really the case and we need to dig in a little more deep, but boy, we really need to start thinking hard about what’s going on here, why these are generating these kinds of results, and whatever the reason why, let’s do it.”

Now, I’ve sort of tried to get into this a bit by looking at not only vouchers and education savings account, but also scholarship tax credits and individual tax credits. And what I’m tentatively getting out of this is that the latter two, scholarship tax credits, individual tax credits, tend to cover a lot more students and families, but with not much funding per student, and ESAs and vouchers tend to be kind of the flip where they cover not many students, at least not in most cases, that’s changing here, but in most cases, they don’t cover a lot of students, they’re usually in with low-income families, disabled kids, so maybe veterans’ kids and things like that.

But the amount that they provide per pupil to people who are eligible is very, very large. And so I’m thinking maybe that’s the real key because after all if you provide somebody with $500, which is some of the individual tax credits, very low amounts, $500, maybe $1000, that probably is not going to induce a lot of shopping around so to speak for a private school for your kids, and then invigorating the private sector that will go after those funds. But if a family gets, let’s say 8, $10,000, now they can start really “go shopping”, so to speak, and be plenty of private schools who are going to be eager to have them as a client.

So I have a sense that’s one of the things that’s going on here is just having some reasonable amount of funding for the kids that are eligible. So I would hope that’s what policymakers would be looking at pretty intensely.

Drew Catt: And it’ll be fascinating just to see what the results might be if replicated for, especially the West Virginia ESA program, once it’s up and off the ground since it does have such a large eligibility compared to other programs, and especially with what happens in Arizona with a push for the universal expansion of their ESA program and kind of see how the effects may change.

So, John, what are your next steps for this research? Are you looking to just kind of tie up what you currently have and get it submitted to a journal or working paper series? Or are you looking to add additional years of data, knowing that things do get a little messy with any education-related data when it comes to what happened during COVID? So 2020 and 2021 data can get a little messy, and 2021 data really isn’t even available yet, since those of us who work with education data, usually you’re working with a two-year lag.

John Garen: Right. Well, that’s a good question. I do want to buff up this paper a bit and sort of get it out there, so to speak, and whoever wants to circulate it, that’s fine. I think there are some follow-on things with this and, obviously, looking in a little more detail in what specific aspects of these programs seem to make the most difference. As I say, I’ve sort of tentatively think it’s the amount per pupil, really, rather than having a lot of people with a little bit of money, you’re better off having fewer people with a lot of money, so to speak.

But I also think that it’s been a challenge and maybe I’ll bug you guys about this going forward, it’s been a challenge to try to characterize state’s choice programs because a lot of the states may have, let’s say, multiple scholarship tax credit programs or multiple voucher programs, and trying to characterize that entire mix of programs is a little bit of a challenge. And at this juncture, my preliminary results, I just take the original voucher program in the state and look at that. But many states have had, particularly states that are sympathetic towards choice, have more than one voucher program. So that’s something I’d like to do going forward is just sort of get a little more, or maybe a lot more, detail about what’s going on in these states that have been so successful and hopefully helping other states replicate that success.

I’m sure you guys know this, but Florida is one of the just remarkable cases where, in the 90s, their test scores were behind the U.S. average and they began implementing choice programs, or I guess around 2000, and then followed on with several other choice programs, not just the one. Their test scores have just jumped up and are substantially above the U.S. average at this point. So whatever they’re doing down there, something is working. And other states have had good success stories as well.

So getting into the weeds, I think, and maybe specific cases about what’s going on in certain states. And I have a sense that once it gets going, it’s kind of in the air so to speak, that once you get choice going and you develop a set of schools that offer different kinds of programs for different kinds of kids and the public schools react to that and say, “Okay, well we need to alter what we’re doing as well.” So I think once that gets going, it becomes part of the atmosphere and it just seems more natural at that point. So I think maybe that’s part of the story, and it would be worth kind of looking at a host of case studies perhaps at the states that seem to have been successful, and comparing that to states that have not.

Drew Catt: Yeah, thanks. And so informational and thank you so much for your time. And it was great meeting you the other month and great to know that there’s another school choice researcher in the kind of greater Midwest area. So yeah, thank you so much for your time today, John.

John Garen: Appreciate it a lot. And I appreciate you guys at EdChoice for doing a lot of the tough work of collecting the data as well as doing good analysis of it. So thank you.

Drew Catt: Hat tip to our listeners for taking the time to learn a little bit more about John Garen and his school choice research. To stay updated on the latest school choice research, legislative news, and more, please remember to subscribe to our EdChoice Chats podcast wherever you get your podcast because we care about choice for more of our coverage of new school choice research, education reform, policy chats, and more. If social media is more your thing, follow us on Twitter, Instagram, Facebook, even TikTok. You can find us @EdChoice, although TikTok, I think it’s…

John Kristof: EdChoice.official.

Drew Catt: EdChoice.official. Thank you, John. Thanks again for listening and until next time, take care.