In today’s EdChoice Chat, Director of State Research and Policy Analysis Drew Catt talks with Jonathan Mills, senior research associate in the University of Arkansas’s Department of Education. Jon talks about his current research projects, what he’s learned from collaborating with a diverse team and how research can be useful for parents and policymakers.
Drew Catt: Hello. I’m Drew Catt, EdChoice’s director of state research and policy analysis. Today, I’m in the studio to introduce our listeners to a researcher to watch. I’m here with Jon Mills, senior research associate in the Department of Education Reform at the University of Arkansas; and former post-doctoral fellow with the Education Research Alliance for New Orleans. Thanks for joining me today, Jon.
Jon Mills: Yeah, thanks for having me out, Drew.
Drew Catt: Jon, would you mind introducing yourself and telling us a little about what attracted you to issues in K–12 education and school choice?
Jon Mills: Yeah, my interest in K–12 education stemmed from back when I was a kid. I had great experiences with my own. Education was an important part of my life growing up. I had … like everyone, we had those key teachers that make you want to go in to consider teaching. Actually, when I was an undergrad initially, I had gone in for engineering; and very quickly had my crisis of, “Who am I?” It really was I wanted to be a teacher.
Over time, I found my way into economics, and I studied labor issues, or specifically issues around education; because I was still very much so interested in how education, the important role in which education can play in shaping individuals’ lives, and creating opportunities for them. For a similar reason, that’s how I found my interest in school of choice research. I study charter schools. I’ve studied public schools of choice. I’ve studied private schools of choice, like the Louisiana voucher program. It’s all kind of been motivated by this interest in creating opportunities for individuals to kind of seek out better matches of their educational needs, and matched with the offerings of schools.
I’ve gotten to see firsthand how in choice environments, schools have to view their … the families as consumers of their education. They have to reach out. I think that that process in and of itself can promote a lot of positive experiences in education that just … Those are the types of things that motivate me when I’m thinking about doing a study, or why I’m in this field in the first place.
Drew Catt: Yeah, so it sounds like the … it’s about finding the best fit educational environment for each student.
Jon Mills: Yes, exactly. Studying systems that can facilitate the opportunity to find the best fit, yup.
Drew Catt: Yeah, and I’m sure within that, it’s very clear how powerful information can be.
Jon Mills: Yeah, obviously. Information plays an extremely important role. As I said, my background’s in economics. In economics, the informed consumer is one of the most important assumptions of that field. One thing that I think is lost on people is that information can take a lot of different roles. Typically, from a policy perspective, we’ll think of information as simply providing information about schools. How are they performing by some metric? Are they moving student achievement? Are parents satisfied with them?
But then also, some of more of my recent research has been looking into the important role that experience, like actually having a one-on-one experience in a school environment, or with some type of mentor; how that can … That is a new type of information. That also is going to fully inform one’s experience, and how they learn to become educated consumers of education; like taking control of their … own educational experiences. Those are some of the things that we like, down at Arkansas, that we’re looking at.
Drew Catt: Yeah, and it’s … Personally, I’ve been a college mentor and a college coach before. Just from that side of things, it was so powerful, just to see the one student that I was interacting with; and kind of helping them from being the first in their family to apply for college, to actually navigate that system, let alone what it would be like for, say, a student switching from a public district school to a charter school, or to a private school through a voucher program. I’m sure that would be a very interesting transition that I’m sure would be beneficial to have someone helping them along the way outside of their family.
Jon Mills: Yeah, yeah, especially … I mean I think it’s a … the link to higher ed is … That’s a great example, because … I have some research where I’m looking at first generation college students. Something that, a lot of these interventions kind of fall into categories. We have financial interventions—let’s give them a scholarship. There are support interventions, and I guess … there’s three categories. The other one is more recently, we’ve seen kind of informational nudges; to getting text messages. You need to move … You need to do these certain things right now, and it’s been shown to kind of help students on the margin persist in college.
One of the things that’s not really being explored is just that importance of when you think about a first-generation college student, one thing that they don’t necessarily have in their world is just someone that they can call and talk to about the struggles they’re experiencing in their first semester in college. I mean all of us, if you’ve gone to college, you realize, this is a different … it’s a different beast.
It was nice for me to be able to call my mom and say, “This is a lot harder than I thought it was, and I’m just trying to make these deadlines. I don’t even really know what to do.”
To hear from her, “Yeah, that was the same for me.” That in of itself, it calmed me down. First generation students don’t have this. Similarly, like you said, students that are taking advantage of choice systems, if their families don’t necessarily have these social networks or things built up, that’s going to produce, one would imagine, initially a struggle. Some think that that struggle is … some fear that that struggle is enough to justify not having a choice system in its own; but others like my good colleague and mentor, Pat Wolf have done some research showing that actually, families, they might struggle initially in this new environment; but like in most situations, they adapt to that environment. They learn to ask questions differently than they used to. It might take time, but they can evolve in that experience. Again, these are the fun things that we are looking at down in Arkansas right now.
Drew Catt: Yeah, that’s great. What about some of your other research, maybe some of your past research or what you’re hoping to work on in the future, what you’re currently working on?
Jon Mills: Sure. I mean the biggest project that I’ve been working on for years now that stems from my dissertation work is this ongoing evaluation that we have of the Louisiana Scholarship Program. This is a private school of choice program in Louisiana that went state-wide in 2012. I’m working with a group called School of Choice Demonstration Project. It’s headed up by Pat Wolf at Arkansas, where we’ve looked at several dimensions of this program. My work is largely focused on how using scholarships, using vouchers, has impacted student achievement over time, but we have a number of papers. We’ve looked at the experiences of students with disabilities in the program. We’ve looked at the competitive effects of the program, so how students in public schools facing increased competition from this choice program, how their achievement was impacted.
We’ve looked a number of dimensions. That project’s been great in terms of we’re … I think that we’re really digging into how this program is affecting not just individuals participating in the program, but the system as a whole. A lot of my … some of my big research interest right now is just delving in further. We’ve done a lot of good causal analysis, so trying to estimate literally how this program has impacted families and students, but now we want to look … We’re trying to dig into what we as researchers call the ‘mediators of the program.’ Basically, why we see the effects that we see? How do the different types of schools that are participating in the program? The private schools, how do their characteristics affect the outcomes that we’re seeing?
We also see that over time, students who have been given a scholarship and use it to enroll in a private school, we see that they have … a good number of them have opted to have switched back out of the program into public schools. We’re trying to document one, what are the characteristics of those students? What were their experiences like? What would have maybe led to that switch out? Then two, follow them over time and see how their educational experiences after that switch were impacted.
Again, if you think about the families as consumers of education, a student leaving the program isn’t necessarily a bad thing. It could be reflective of somebody actually determining that “this educational environment is just not for me, and the public schools are a great option for me.” We’re looking into those things, and that’s … I think it’s pretty exciting stuff.
Drew Catt: Yeah, and I wonder if the proliferation of the charter sector in New Orleans played an effect at all in that.
Jon Mills: Yeah, so we’re focusing on the statewide program itself. A lot of our … like if you’ve seen our achievement effects analyses, a lot of those are actually driven by families outside New Orleans.
Drew Catt: Okay.
Jon Mills: However, we have done sub-analyses that have looked at the experiences of students in New Orleans. It is true that actually the effects are more negative for students using scholarships to go to private schools in New Orleans. When we dig in deeper, it looks like that’s because the control group students, students who stayed with the public option. In New Orleans, it’s all charter schools at this point. They actually were experiencing even larger gains. My old colleague at Education Research Alliance, Doug Harris, has looked at this. Yeah, he’s demonstrated that the post-Katrina reforms have led to markedly positive effects for students. Sometimes, or if you’re … certainly if you’re … if the control groups has a very good educational environment, or an environment where there’s rather demonstrated growth, which is what we seen there, that’s why we see these negative effects for private schools in that case.
Drew Catt: Yeah, and these effects are still on the achievement of these standardized tests. Is there anything in the future looking at attainment, whether that’s graduation rates or college matriculation?
Jon Mills: Yes. It’s taken time to get a sufficient sample size to be able to run this analysis; but one of our grad students, Heidi Holmes down at Arkansas, is leading up a portion of the evaluation that’s going to be looking at high school attainment and then specifically, now we’re getting in data on college enrollment. Basically, the majority of students who apply to the program in that first, in which we have lotteries, and that’s why we can study these things causally; the majority of students who are applying for grades K through 5, but now because we’ve tracked enough years, we actually have enough students that could have gone to college.
Yeah, I mean these data, we’re just getting in now. So excited over the next year to be … hopefully that’s something for everybody’s radar is that we’ll be releasing some research in the next few months that’s looking at how this scholarship program has actually affected attainment and college matriculation, yup.
Drew Catt: Yeah, that’ll be really interesting to see what those results are like. So Jon, what is it like for you within the university setting to collaborate with other people in your department, or with other universities? What’s that experience been like for you?
Jon Mills: Oh, it’s fantastic. I was lucky at Arkansas in that our department has … We’re all quantitative researchers, but our backgrounds are in different areas. This was also true at Education Research Alliance for New Orleans. We had people who had studied … Their training was in public policy. We had psychologists. We had economists. At Arkansas, we also have a lot of poly-sci folks. I love being in that environment, because you get very different perspectives on the same type of intervention; which I think generally gives a much better and more … it allows for more comprehensive looks at interventions.
Similarly, mixed methods research; so this is research that includes both quantitative elements and qualitative elements. Actually, not only just looking at … Like my job is basically to hopefully provide a good 30,000 foot perspective on how a program’s working. It’s great when you get the opportunity to combine that research with qualitative researchers who are going into schools and actually trying to do interviews with people, to really kind of flesh out how these programs are impacting families. In my ideal world, that would be every research team that I work on is going to be a combination of both groups; because again, our big thing with just policies in general, certainly in education, is that they’re typically designed to try and achieve one intended effect. But almost always, if you introduce something new, there’s going to be a ton of unintended consequences. Some of those can be very good consequences. Some of them can be bad consequences.
When you … That’s like the goal of doing a … If you’re going to do an evaluation, you want to do it well and comprehensively to document what did go as expected, what did not go as expected, and how, if people were trying to replicate this, what they should expect. Yeah, the academic environment’s great, because we get a lot of … I get the opportunity to do that.
Drew Catt: Yeah, and in terms of replication, how important is it to you to have data that you’re producing or understandably, it’s if you’re getting a data share agreement with the Department of Education, it’s typically restricted use and you have to strip out all the personally identifiable information. How important is it to you as a researcher to work with data that you could potentially share with others, or looking at someone else’s data set in order to be able to potentially replicate the work that they’ve done?
Jon Mills: Yeah, replication’s extremely important. When we think about advancing knowledge, about how programs work, or just generally, the pursuit of truth, if you relied exclusively on one study, or one study of a single intervention, there’s a good chance, so in statistics, we literally build in the possibility that you might find a significant effect. You might conclude that a program has an effect when it happened due to random chance.
If we want to know how these programs work, the extent to which we could pick one up and put it in a different area, we have to fully understand the context of the environment that we’re currently working with. You also have to understand how these programs might work in different areas. That’s what … like replication is key. You have to be able to see over time if these effects actually pan out or if they’re actually just due to random circumstances. Yeah, in terms of just … I think for all researchers is you want to be building your data in a way that somebody else could pick it up and run it, because … that’s another thing is that other people might have different perspectives. A lot of these analyses are built on assumptions. It’s always helpful when you have somebody else point out like, ‘Oh, maybe this assumption actually doesn’t seem to work here.’ They might have a slightly different take on it. If that produces different results, then we might be concerned, right?
Drew Catt: Mm-hmm (affirmative).
Jon Mills: We want to know that so we can move forward.
Drew Catt: Yeah, which is why I always enjoy going to academic research conferences; which I believe we initially met at one of those Association for Education, Finance and Policy a few years ago. It’s great to get that push back from other researchers who are coming at the same issue from a completely different angle, or looking at it through, as you were saying, with the different backgrounds of faculty and staff at the university, looking at the same problem through a different lens. Yeah, so I always personally enjoy hearing other people’s perspectives because I think it helps our research become stronger.
Jon Mills: Yeah. We shouldn’t be in the business of satisfying our own egos. To me, what I try and remind myself and my students when I’m working with them is that what we’re doing affectively is making an argument. Statistics are a part of our tools to support that argument. They should not be afraid. I shouldn’t be afraid of engaging with other researchers. I should actually … It’s extremely important that we do engage with other researchers to see how strong our argument actually is. I might be convinced of what I’m doing and what I’m finding, but I want to know the extent to which I can convince somebody else; because … especially with policy generally, but in education policy, we’re dealing with world of scarce resources.
If you went out to academia and asked … polled every education policy researcher and said, “We’ve got X amount of money. We want to invest in a program. What should we do?” I’m going to think that unsurprisingly, that program that they propose is going to look a lot like what their research is showing. Over time, we have to think about like when distributing scarce resources, it really needs to come down to the strength of your argument and the strength of the evidence, rather, supporting these different things; because the policymakers, they have to make the tough decision of how do we divide up these resources?
Yeah, that’s … exactly, like you … This is one of the big benefits of academia is that we get to, on a daily basis, I get paid to think and I get paid to engage with other people who are telling me I’m wrong. I get to try and learn from those experiences.
Drew Catt: Yeah.
Jon Mills: Yup.
Drew Catt: Yeah, no. That totally makes sense. For example, a legislator looking at enacting educational policy in West Virginia would probably tackle it a little bit differently than a legislator in D.C., or in the Louisiana area, or Indiana, because their community has potentially different needs, and looks a little different.
Jon Mills: Oh yes. Yeah, yeah, exactly. It is also funny, though. I do exist in the ivory tower. It’s always interesting when you have … I think sometimes people forget that legislators face, like you said, they’re reacting to their local constituency, as they should. That’s going to have certain political factors in which programs that might make tons of sense to an academic, just why it would … it would literally … They would serve no purpose for that locality. This is, to me, a natural thing and a good thing. It’s another reason why I think it’s very important for researchers of policy to really be acting and engaging with legislators, just to get a sense of why things would or wouldn’t work. Often, those conversations don’t necessarily happen. I think we could all learn a lot from those conversations.
Drew Catt: Yeah. Any advice for any of the listeners out there that are interested in school choice, but haven’t necessarily made the deep dive into doing any research themselves, or potentially interested in going back to school to learn more about education policy?
Jon Mills: Oh yeah, I mean if you’re on the verge, just jump in. There are a lot of great resources out here. The work that you guys are doing here at EdChoice is a nice place to start. Your reports are great, very easily readable, and can give you a broad sense of the way … Like if you’re specifically interested in private choice options, there’s a lot of resources there to check into. Yeah, moving for … Again, it’s just kind of … If you’re asking yourself that question, why not just dive in and you’re probably not going to … you’re probably not going to have much trouble.
Yeah, and of course, if you’re interested, if that initial jump into learning about private school of choice gets you really interested in pursuing a program, I’d always recommend looking into our Ed Policy Ph.D. program at the University of Arkansas. Reach out to Pat Wolf or myself, Jon Mills.
Drew Catt: Yeah, that’s great to hear. Any last words? Any forthcoming research you’d like to plug other than what we’ve already discussed?
Jon Mills: Yeah, I think the big thing is with, like I said, the … We’re trying to provide a very comprehensive look at the how the Louisiana scholarship program has been affecting families and the education system as a whole down in Louisiana; which is important both for general truth and also specifically for policymakers for those families. I’m just very excited about the upcoming studies. We’re going to be looking at attainment. We’re going to be continuing our look at achievement, and diving into how … just some exploratory descriptive studies that are looking at what features of these programs might be explaining the results that we’ve seen over time.
Drew Catt: Yeah, that all sounds great. Well Jon, thank you so much for sharing some time with us today.
Jon Mills: Yeah, it’s been a pleasure. Thanks for inviting me.
Drew Catt: Yeah. To our listeners, be sure to subscribe to our podcast for more of our coverage of new school choice research, education reform policy chats and more. Thank you for listening, and we’ll see you soon for more EdChoice chats.