What does it mean to hold schools accountable? We convened focus groups of K–12 stakeholders—from policymakers to researchers to teachers—to wrestle with that tough question and help us better understand the current accountability landscape. In our latest report, Do-Over or Double Down? Working Toward a New K–12 Education Accountability Ecosystem, our researchers compiled what those participants said.
Download the full report or listen to the authors, Mike McShane and Paul DiPerna, discuss their methods and findings.
Paul DiPerna: Hi, welcome to the EdChoice Chats podcast. This is Paul DiPerna, I’m the vice president of research here at EdChoice, and I’m joined on the phone today with my colleague, Mike McShane, who’s our director of national research.
So, Mike, our teams have been throwing it down against each other over the last few days. Your Chiefs got the best of my Steelers on Sunday, and then last night, my Pirates actually for once came back against your Royals. I wish I could say these games have some serious post-season ramifications, but alas, only wishful thinking. At least on my end.
Mike McShane: I didn’t realize until the game started, that on my fantasy football team, I had Antonio Brown, the Steelers kicker whose name is escaping me right now, and Steelers defense. So I was in a packed bar watching the Chiefs game and kind of quietly saying, “Well, if the Steelers are going to throw the ball, could it maybe go to Antonio Brown?” Or, “Look, a field goal here and there doesn’t do anything.” Holding the Chiefs to a low number of points, as long as they still win, I think everybody can win. But both the kicker and the defense performed terribly, which while good for me and my compatriots, just absolutely devastated my fantasy team. So maybe, in the end, the Steelers get the last laugh on me.
Paul DiPerna: And I have the Chiefs running back on my fantasy team, so I was a little bit of a similar boat. Not at a bar, like you were, but yeah, you guys came out on top. But okay, we’re not here to talk about our KC and Pittsburgh sports teams.
Mike McShane: As much fun as that would be, as much fun as that would be.
Paul DiPerna: That could be a whole other podcast series for EdChoice.
Today, Mike and I are going to discuss our new EdChoice report, Do Over or Double Down? Working Toward a New K–12 Education Accountability Ecosystem, which represents the culmination of a lot of input and shared perspectives from a lot of gracious and talented people from all across the country.
Mike McShane: Yes, this really was a labor of love, both on our part and all of the people that participated in it. So it probably makes sense to start at the beginning. Paul, how did this report come to be? What inspired it?
Paul DiPerna: Yes, this goes back a couple of years, where really a lot of credit goes to Brian McGrath here at EdChoice, who pulled me aside. I remember we had a meeting, just trying to think about a different type of project for EdChoice that was research-based, empirical to some degree, and to try to utilize focus groups or gathering people in some way. So we really were thinking about the design and the approach first, and it’s really to Brian’s credit to get the ball rolling with that.
And then we had conversations with Robert Enlow, who’s our president here, and we were thinking about what topics could really be well-suited for some type of expansive focus group project. So we thought with some things really happening in the news and just developments in the policy world around accountability, that this could be a really good time to explore accountability very broadly, because we’ve seen it within our area of work in school choice, but certainly accountability has a broad reach affecting various types of activities and reform efforts at the state and local levels. With ESSA’s enactment, the Every Student Succeeds enactment a couple of years ago through Congress and signed by President Obama, we thought that there would be some changes happening in the states, and there could be just a really good opening for some exploration and truthfully, some reflection, before implementation really starts moving quickly in the states.
So there was the ESSA enactment and everything that came with that, and then also at the same time, and we’ve seen this even going further back where there’s some years-long pushback or at least some fatigue when it comes to state testing for purposes of accountability. So that has continued to be in the news, raising legitimacy questions and feasibility questions, certainly moving from paper and pencil to online and computer-based testing. So that also is kind of another interacting factor. And then finally, we just thought that with all of our work and research on school choice and ed reform issues, we really, from our organization’s perspective, wanted to see what the future of K–12 accountability could look like on at least two levels. Very broadly affecting a whole range of folks engaged in K–12 education activities, such as teachers, students and parents, families, schools and the larger governing structures at the state level.
And so all those things combined, I think, are really a nice opportunity for EdChoice to put together this project and really seek out larger perspectives outside of school choice and outside of those who would be considered as “experts” on accountability, but really to try to look to others who are familiar with the issues, maybe not engaged in them in a daily basis, but really to get their perspectives and just get their points of view on how accountability has gotten to where it is today.
And the finally, we really should make a special note to thank the generous financial support of the Walton Family Foundation, who supported this project. Without their generous support, we would not have been able to do this type of project, especially at the scale that we were able to pursue.
So that’s a little bit of background regarding the project and how it came to be. Mike, maybe, would you want to take a couple minutes to say who are the audiences for this type of report, and who we were trying to speak to with this project?
Mike McShane: Sure, I think we were really trying to have a broad audience for this project. Obviously the report that comes out of it, this podcast that we’re listening to, other things that we’re going to do around it, because everybody has a stake in school accountability. I think that sometimes that’s lost, we oftentimes focus just on schools and teachers and how they intersect with school accountability systems. But fundamentally, school accountability systems are about what we as citizens want and expect from our schools. So I think that it’s really important that everybody realizes that they have a role to play, a stake, an interest in accountability policy, and so their opinion matters. It matters what parents want from schools, it matters what teachers and professional educators think that children should learn and when they should learn it and how they should learn it, and also the broader body politics matters, because they have expectations, they’re the ones that pay for all of this. So what they think is important.
So hopefully, by also the way that we structured our focus groups, by trying to draw from a broad cross section of the education community, we had a lot of different viewpoints represented, a lot of different stakeholders with their own kind of motivations and responsibilities. So hopefully something within there will speak to everybody.
Paul DiPerna: Yeah, I totally agree. I mean, that was a big thrust. We really wanted to make this kind of a big-tent type of project, to really speak to and listen to, really, as many different perspectives as we could. And I think, would you say, Mike, this is really a different type of research project compared to what we’ve done here at EdChoice in the past, where a lot of times we do large end studies or quantitative polling surveys types of research projects. This is a little bit different.
Mike McShane: Yeah, absolutely. We have a strong tradition of great polling that we do both with the national and the state level, and both quantitative and other forms of qualitative or sort of “though leadership” stuff. But we haven’t done a lot of these focus groups. So for me as a researcher, it was fun. It was definitely a challenge, because I think in the end, we had north of 30 hours of video of these sessions that took place. I think the transcripts lasted more than 700 pages. So trying to bring those ideas together into some kind of coherent form was definitely a challenge, but maybe it’d make sense, Paul, if you do a little bit of the method. Because I think we’ve been speaking kind of generally about what we’re doing, but if you could walk through exactly how we did what we did, and then we’ll get to how we made sense of all of that.
Paul DiPerna: All right, we’ll get back to that labor of love that you referred to earlier.
Mike McShane: Yeah, exactly.
Paul DiPerna: Yeah, so just to give a little bit of an overview of the approach that we took, EdChoice hosted two back-to-back convenings of what we would consider folks who are influencers and connectors engaged around K–12 education generally, and we hosted these folks just outside of Fort Meyers, Florida, back in April, the last week of April. The total substantive programming lasted more than eight hours, and what we did is, we had four focus groups at each of these convenings, and each focus group roughly had six participants. We had a couple people who had to cancel, as that kind of thing happens, as we got closer to the event. But generally six per focus group, and then these focus groups, at each convening, they were facilitated by Hanover Research, our partners at Hanover. They led these focus groups.
So basically, the run of the couple days for each of these convenings went like this. In the morning, we had you, Mike, who did a terrific job giving a kind of primer on accountability, which gave a historical lens to understand how we’ve come here and just a timeline of major events and developments when it comes to accountability in K–12 education. So that kicked things off, and that assured that we had a kind of common ground where everybody would have some common knowledge and shared knowledge about K–12 accountability. From there, we broke out the participants into their respective focus groups, or in the report, we also called them affinity groups, because each of these groups were organized in a way around where these folks are coming from and their professional backgrounds.
We had four types of affinity groups. We had engaged outsiders, who could be folks from the business world and private sector, people from the US Chamber of Commerce. We also had others from non-profits like Great Schools, and so they had this kind of outsider perspective. We also had advocates, policy advocates, who are folks who do a lot of work at the state level. We had practitioners who would be current or former teachers, also a principal, a superintendent, and then finally we had researchers and scholars from mostly academic settings, some people who were coming from think tanks. So based on those four general affinity groups, then they broke out into group discussions led by Hanover, and there were a series of sessions where we were asking them about how have we arrived at where we are today when it comes to accountability. And then we asked them questions about the current state of accountability, and then we asked them to do a simulation.
And maybe, Mike, you might want to go into this a little bit, how the simulation went. But we asked them to be given a set of circumstances where they were responsible for designing a type of state-wide accountability system for a hypothetical Midwestern state, and how would they approach it. Just basically giving them a blank slate, what would they come up with for what they would see as a legitimate and useful state-wide accountability system.
Mike McShane: Yeah, it was a really interesting process. I mean, I think it made sense to, we started by looking backwards, and then we sort of came … So what lessons have we learned over the last 20 years of accountability policy, then we brought it to the present and had a nice discussion around, “What is the current state of accountability?” And then we tried to look to the future, both in the simulation where, yes, their groups were asked to advise a fictional Midwestern governor, if they could design a school accountability system for that state, what would it look like. And then also, the series of large group discussions, where in each case, over the course of it I think we had 47 participants, so in the two large group sessions, they had an opportunity to speak kind of across their affinity groups, which was a fascinating process because it turns out, oftentimes, researchers and practitioners don’t exactly see eye to eye on what the priorities of an accountability system should be, or others.
So it was a really interesting mix of, again, we’re super thankful for the folks of Hanover Research, who I think did a great job as facilitators and helped really kind of push and prod and get information out of folks. I do think it yielded a lot of really interesting information.
Paul DiPerna: They did do a wonderful job. I thought it was interesting too, how these different affinity groups, and correct me if I’m misreading this, Mike, but it seemed like the researchers in both convenings, they kind of gravitated towards looking at questions around measurement and what’s valid, what’s reliable, what’s fair, what’s useful, what’s worked in the past, what looks promising right now. And then you had practitioners definitely looking at it from a kind of implementation perspective and how things get implemented and can be designed to a T, but then once it gets to school level and classroom level to carry things out, things can really change, the game plan can change. And then some of the other groups just really came from different points of view.
Mike McShane: Yeah, absolutely. So if we look at the high-level things that I think that we learned from this, I think there was much more agreement kind of looking backwards. There was more agreement about some of the lessons that we’ve learned, and I think a fair amount of agreement about the current state of accountability. I think you’re right that as we look to the future, though, different people have different priorities. So I think if we look at the lessons that were learned, what we tried to glean from these conversations, what we think the big takeaway, the big lesson of the last 20 or so years of school accountability policy, I think the folks sort of focused on two things. The successes of the accountability movement and some of the challenges of the accountability movement. They were clear that they thought that there was success, or has been success from the accountability movement in the focus and attention in equity, in promoting greater transparency, in helping schools and families and communities become more data-literate, and also that there was an improvement in test scores as a result of accountability.
Now, that said, they did talk about some of the missteps or the challenges that they’ve had. There has been, I think a pretty shared consensus, not a unanimous one, that accountability policy has narrowed the goals of schooling, that it has incentivized gaming, it’s incentivized drill and kill strategies that sort of maximize test scores at the expense of real learning. I think another thing lots of folks coalesced around was the belief that there’s sort of school accountability, and then there’s teacher evaluation. And when teacher evaluation became part of, kind of under the umbrella of accountability at large, that was a serious problem. A lot of the pushback, I think our participants argued, a lot of the pushback that accountability sees now really, the sort of gasoline thrown on that fire was deciding to try and wrap teachers into that as well.
And so when we look at the current state of school accountability, I think there’s sort of four issues that people highlight. One is a lack of clarity of purpose. Why do we have accountability systems? What is the goal? What are accountability systems trying to do? The second issue that’s taking place currently is this idea of layering. That there are federal demands, there are state demands, and there are local demands. In many cases, there are assessments aligned to each of those demands, and this kind of patchwork layering of different levels of accountability, so schools are accountability to the feds, they’re accountability to their state, they’re accountable to their local community in sometimes intersecting ways. It’s incredibly frustrating for people, it’s unclear and it’s troublesome.
I think the third thing that folks talked about with the current state of accountability is that we’re not measuring what parents care about. People want a lot more from schools than simply reading and math scores. The kind of ways that we’ve tinkered around the edges, whether it’s looking at graduation rates or others still are not giving people the full picture of what they want to know about what’s going on in their children’s schools. And then finally, it’s impossible to talk about the current state of school accountability without talking about the politics. It’s clear that things like standardized testing and others are increasingly unpopular. I mean, you saw with the example of the Common Core standards and the incredible controversy that was around there. Again, things around teacher evaluation and others, that accountability has a political problem. Standardized testing that it’s based on has a political problem. So any conversation about accountability moving forward has got to wrestle with the politics of it.
So I think those were, again, from lots of different folks’ perspectives, but that there was relative agreement. Again, not universal, but a fair degree of agreement around both that kind of past and present of the school accountability movement. It got more complicated, as one might imagine, as we look to the future. It’s a lot easier to diagnose problems and to maybe share these things when looking at the future. But I will say that I think we were able to glean some ideas about how to move accountability policy forward. We identified them as sort of new priorities and new data, and some new conversations that need to take place.
So the new priorities, the number one thing, it appears, that accountability proponents or states that want to have robust accountability systems need to do is to restore trust. There’s an incredible amount of distrust with families, with communities, and particularly with educators, that if you believe in school accountability, restoring that trust, getting people to believe in accountability systems, to believe that what they’re measuring matters, believing that they’re being used in productive ways, is incredibly important. Policymakers, and again, accountability advocates, need to put in the front of their mind, “Is what we’re measuring what matters? Are we using that data in productive ways? And if we aren’t, we need to change tacks, because that fundamental lack of trust is going to undermine everything that we want to do.”
We also said that there needs to be more consideration and honesty about trade-offs. That school accountability policy presents a series of trade-offs, whether those are trade-offs between uniformity and diversity, when it comes to the offerings that are available, when it comes to setting high bars versus low bars, all of those things involve a series of trade-offs that we have to be honest about. I think that there was too much kind of magical thinking that took place that accountability policy is going to be all winners and no losers, or the sort of losers in the system would be dramatically, that was sort of dramatically downplayed. So we need to return to honesty about trade-offs, and I think it’s also true to be much more focused around forming human beings. That education is more than about sort of a narrow set of academic skills. It’s about civics, it’s about really forming human beings that want to be not just productive in the economy, but in society.
Paul DiPerna: Right, good citizens.
Mike McShane: Yeah, good citizens and live fulfilling life. So then as a result of this, we also talked about, in the report that we have, we won’t belabor all of it now, did a long conversation about new sources of data. So it’s like, if we want to collect better data on these more robust things, there’s a whole list. I mean, I think we bring forward 25 or 30 different things that could possibly be measured. Obviously all of these things have their pluses and minuses and would need to be discussed, but we wanted to have it be productive and sort of surface these things.
And then we closed with some debate about all of that. But look, Paul, I think of you as a very fair-minded person.
Paul DiPerna: Why?
Mike McShane: Even though you are a Steelers fan, I do think that you are of sound mind and judgment, but I think it’s important, we’re always trying to be conscious of what maybe critics might think. So I’d be interested, what do you think, when the paper is published, what do you think a critic might say about our paper?
Paul DiPerna: Yeah, I think that’s something that we keep in mind, of course, when we’re doing this type of writing and analysis. Just to anticipate any of those kinds of questions. I think it’s like, who did we have in the room. Whether it was enough teachers or enough principals, people on the ground, practitioners. Did we have enough folks who look at measurement?
So I think it could be something around that, how rigorous were we in terms of trying to bring different perspectives in these different affinity groups together. And it was something that we really tried to be as methodical and deliberate as we could in terms of inviting these folks who, really, I mean, just blew me away for the one convening that I attended and observed. Just really, the open-mindedness that people brought, they were there to share their opinions, but that was something that really struck me how people were willing to listen and wanting to learn. We were there to learn. Now, hopefully, that comes through in the report. Mike, I mean, you did a lion’s share of the writing and the analysis. Going through those transcripts and the videos, I mean, it was really a yeoman’s effort and terrific work that you did to lead up to this. So we really tried to leave no stone unturned and be as rigorous as possible with this type of focus group series, which was a new thing for us too.
Mike McShane: Yeah, and it should be clear that we in appendices to the report have the discussion guides that we used, the questions that we asked, all of that. So we were as transparent as possible in how we framed things, how we asked questions and all of that. So folks, as always, with all of our research, we encourage people to check the math. To go in and see what we did and then base their judgments accordingly.
Paul DiPerna: Yep, that’s right. We have those discussion guides in the appendices and try to have as much of that information out there as possible. So Mike, getting close to wrapping things up, I was thinking about those audiences for this project and the report. If there’s one takeaway from this overall project, I mean, what did you come away with?
Mike McShane: Well, for me, I mean, I would like to … I’m generally, I think a kind of optimistic person who looks toward the future to try and solve problems and others. I think one of the sections that we kind of conclude with is looking at these new conversations. A lot of the questions of accountability are unchanged, these fundamental questions. But sometimes I think we have either waved our hands at them or alighted them, and I think that we’re just not going to be able to do that anymore. We’re going to actually have to get to the root of some of these competing questions. So some of these questions that we close the paper with that I think we need to talk about and we need to talk honestly and forthrightly about, some of them are like, “Is the point of accountability systems to establish a minimum standard or to drive schools to be better?” So looking at accountability as minimum standards versus driving excellence.
I think we have to be honest about the trade-offs that are included in that, the assumptions that are included in that, the uniformity that may be included in that, and be honest about what’s happening. Another question is A through F grading. Should we try and roll up all of these disparate measures that we have in one thing, so whether it’s A through F grades or whether it’s some sort of performance index, what are the pros and cons of trying to do that? I think, can we have strong accountability without enforcing uniformity? Again, weighing these trade-offs between having something like a uniform A through F grade or performance metric or whatever, and the lots of different things that schools want to do.
A fourth question that we definitely need to wrestle with is, “Do we trust parents to know more than the school system?” And I think we sort of put our cards on the table as big school choice advocates that we do, we trust parents. But we also recognize that not everybody thinks the way that we do. And in order to have some kind of accountability system, particularly accountability system in a system of schools that has a lot of school choice, we have to weigh out how much do parental preferences express in the types of schools that they choose. How do we weigh that against some of these objective measures that we think that we value and that we would like to collect?
And then finally, there is this big question about what is the unit of analysis? You know, we’ve talked about generally speaking with accountability, the unit of analysis is the school. But then with teacher evaluation, we said, “Oh, no, it should actually be the teacher.” But with a lot of stuff that’s happening around personalized learning or others, is the actual unit of analysis the student? Do we care about things, again, in a more school choice-rich world, do we care much more about fit? That is to say, we may have a school that, on average and in aggregate, performs highly on a set of metrics that we care about, but there may be students that will not thrive in that environment. So it may not be what we call “a great school” might not be a great school for everyone. How do we think about that and how do we measure these really complicated notions of things like fit?
So hopefully, what I hope people take away from this is rather than using this to kind of man the ramparts of their particular views about school accountability, use these insights to think about some of these fundamental assumptions that we have, to think about these questions that still after 20 years remain unresolved. And maybe question your own assumptions. I know certainly in having these conversations and listening to these folks, it caused me to question some of the things that I think and hopefully get a much finer tuned opinion about some of these things. That’s what I hope folks will take away from it.
Paul DiPerna: Yeah, likewise. I definitely, just talking with different people, I was learning things from different perspectives. Your point about questioning current assumptions, my own, and then also I think the way policy making has kind of gone forward the last 15-20 years, it seems like, going back to what we were talking about earlier, about this being a window of opportunity as ESSA starts to roll forward and implementation plans are being proposed at the state level, it seems like we have this window right now for the next year or two, to really step back and to question those assumptions and to be open-minded to these different stakeholders who have been affected through accountability policies and to see if maybe there’s a different path to go on.
And along those five different questions that you laid out about these kind of new conversations that we can have, so that’s really my takeaway as well.
So, what do you kind of expect to come from this report, or hope to come from this report after it’s released and kind of percolating out there? Do you see any follow-up, or how could people from these different types of affinity groups or other perspectives be using this type of report?
Mike McShane: Sure, it’s tough for me, because I tend to view things from the perspective of a researcher. So maybe that’s, my bias will be showing here, but one of the things that I took away from this, a great opportunity, whether it’s a young enterprising researcher that needs a dissertation topic, or other folks out there that are interested in this report, we lift this incredible spread of data that folks thought could be useful in better understanding how schools are operating in some places, schools are collecting in them and other places that they aren’t. But I think there is a wealth of research to be done looking into those metrics. What do they actually tell us, what are the ways to measure them? Are there better and worse ways? Are there some things that frankly just can’t be measured? Are there tools that we’re using now that are bad?
So that section on the kind of new data moving forward, I think would be ripe for tons of interesting research, tons of interesting conversation, because we just don’t know, for a lot of this stuff, we just don’t know yet. We don’t know how to measure it, or we have very, very basic tools to measure it now that could conceivably get more sophisticated. We don’t know exactly how it’s used, all of that. So that, to me, I think there are multiple life-long research agendas in those lists of data that we brought up. So I just think there’s a whole wealth of possible research lines that are in there, and we’d be more than happy to try and be helpful in any way that we can. So if there’s a researcher that’s listening to this podcast who would like to spend some time thinking about or talking about these things, shoot us an e-mail, give us a call, we’d be more than happy to chat with you about ways to think through this.
Paul DiPerna: Yeah, I agree. I mean, there is a lot of potential research questions to be examined and investigated, and hopefully this can provide some kind of a road map for grad students and Ph.D. students doing their dissertations or researchers who are early on in their careers. Along the lines of a road map, I think one thing I see that could hopefully come from this report, and that this just really generates those questions, I mean, it’s like that last part that you were talking about. To hopefully encourage people to think about these kind of questions and about challenging assumptions, because we’ve been on a kind of glide path for a little while when it comes to accountability until the last few years with the Every Student Succeeds Act changing things up a little bit. So hopefully, this moment of a pause gives parents, teachers, principals, hopefully they have the chance to maybe pursue one or more of these types of questions, especially those who are in policy-making and decision-making positions, and to kind of hold them accountable.
Mike McShane: Absolutely.
Paul DiPerna: Well, folks, I think that’s all for this episode of EdChoice Chats. You can download the new report at edchoice.org/AccountabilityEcosystem, which is a little bit of a mouthful, but it’s not too bad as a URL.
Thanks to all of you, our listeners, for tuning in. Just a quick reminder that you can subscribe for more EdChoice Chats on platforms like SoundCloud and on iTunes, and we’ll catch you on our next podcast. Take care.