Danish Shakeel discusses his co-authored report, Changes in the Performance of Students in Charter and District Sectors of U.S. Education: An Analysis of Nationwide Trends.
Jason Bedrick: Hello and welcome back to EdChoice Chats, I’m your host, Jason Bedrick, director of policy at EdChoice, and this is another edition of our Big Ideas series. Today, I’m excited to be joined by Dr. Danish Shakeel, a post-doctoral research fellow at the Program on Education Policy and Governance at Harvard University’s Kennedy School of Government. He is the co-author with Harvard’s Dr. Paul Peterson, of a new study titled, Changes in the Performance of Students in Charter and District Sectors of U.S. Education: An Analysis of Nationwide Trends, which is the subject of today’s conversation. Danish, welcome to the podcast.
Danish Shakeel: Hi, Jason. Thank you so much for inviting me.
Jason Bedrick: It’s a pleasure to have you. Before we dive into what you found in the study, perhaps we could just briefly cover what charter schools are. There’s a lot of people who think that charter schools are private schools, or that they can be religious schools, or that they have test requirements to get in. So just what are charter schools, and what makes them different from traditional district schools?
Danish Shakeel: The public education in the United States is divided into two sectors. The first sector is the district schools, and the second sector is charter sector. These charter schools were formed in the 1990s, and then they grew up. This sector receives most of their funding from government at the local state and federal levels, and the district schools are governed by school boards or other official governing bodies, and contrast charters are going by non-profit organizations, which are authorized by a public agency. So they are public schools in all regards, but they have some autonomy in comparison to their public counterparts.
Jason Bedrick: Okay. And your study compares these two sectors, it compares the traditional public school or district school sector and the charter sector using the NAEP, the National Assessment of Educational Progress. So how did you make this comparison between the two sectors and what did you find?
Danish Shakeel: So, we make comparisons of the gains for each cohort of grade four and grade eight across the two sectors. So our study is not comparing the two sectors at a single point of time, we’re looking at overall trends, over a 12-year period. So we start with the 2005 baseline year for which the charter data is very strong, and then we go to 2017. So each two years, the NAEP administers the survey in math and reading to samples of grade four, grade eight and grade 12. The grade 12 data is not very good in terms of numbers so we ignore that, and we only look at math and reading for every alternate year, from 2005. The data which we use is restricted, which means that we have background level variables and it is very detailed.
We also have socioeconomic status, which we measure by the number of books in home or the computer availability in home, and the student-reported parent education level for grade eight. The SES data is not available for grade four. So we use this data and the data is available up to 2017. After that, they have not released it publicly yet. So we compare target trend and then we do our analysis.
Jason Bedrick: Okay. So, to be clear, you’re not just doing a comparison of charter schools to district schools? What you’re looking at is, students that are in each sector separately, how are they gaining over time? And then you’re comparing the gains from one sector to the gains of the other sector. Is that accurate?
Danish Shakeel: Yes. And we did that for a reason because management might think that one sector is growing and the other is not growing, so we show that each of them is growing and then we take the difference so we can show that each of them is growing and the difference is this much, and then we interpret the difference.
Jason Bedrick: Critics, though, might point to the selection problem, right? As I alluded to earlier, charters are public schools, that means they’re not private. They’re not religious. They cannot have admissions requirements. You can’t say you have to take a test in order to get in. If they’re oversubscribed, they have to have a lottery. Nevertheless, parents have to affirmatively opt in, and the types of parents who do that may differ in importance, but unobservable ways relative to the parents whose children are enrolled in their default district school. And so if you think that parenting matters, which I think most people do, if these two different types of parents are different, then that may affect their children, right?
So, in other words, you may have families that are more interested in education, actively searching for charter schools. Those families that the parents are less interested in education are more likely to just default to the school that they’re assigned to, and so what you’re picking up really is not the difference between the performance of the charter schools. What you’re picking up is just the selection bias that these parents are choosing this sector over that sector. How do you address that problem?
Danish Shakeel: Our data does not allow us to track students over time, so that is a limitation of the data. We only observe one student for a particular year when they are tested and we cannot track them over time. So we cannot account for selection and we also cannot account for their prior test scores. So, we don’t do a value-added analysis because of that reason. Now, what we can do with the data is we can control for changes in the student characteristics. So, we can pull it for gender, ethnicity and SES at graded, and despite controlling for those covariates, the difference in gain does not shrink away. So we can say that at the best, with this sort of analysis, we don’t find the differences to be explained by background characteristics. Now, we can support that with evidence from other high quality randomized studies, which have found that despite accounting for selection, the charter effect remains in certain contexts. So our study is not challenging that finding that charters do well in certain contexts, but we show the heterogeneity by different categories to further discuss that pattern.
Jason Bedrick: Dany, another thing that critics of the study have raised is that you may be oversampling charter schools in urban areas and that may be biasing your results. What do you say to that?
Danish Shakeel: That is a very good question, Jason. So what we can do is we can look at two surveys, one is the NAEP, which we are already using, and the other is the service collected by NCES for charters generally.
Jason Bedrick: Right. That’s the National Center for Education Statistics, which is a part of the department of education.
Danish Shakeel: Exactly. So if you look at the two surveys and we look at the urban share of charters in the two surveys, we find that there is no evidence of systematic oversampling of charters from urban areas in NAEP. For example, in the year 2017, the urban share of charters in NCES was 56 percent, in net, it was 52.5 percent, all right, closed a little bit lower. In 2015, the NCES share of urban charters was 56.5 percent, in net, it was 55 percent. Similarly, in 2007, it was 54 percent in NCES and 55 percent in NAEP. So we don’t see any evidence of systematic oversampling as far as the data is available to us. And I don’t know of any better source to make comparisons of NAEP with a national level dataset.
Jason Bedrick: So what do you find overall? Who benefits? Is it across the board? Is it certain subgroups?
Danish Shakeel: So we have three principle findings, one is definitely that the overall gains in charters amount to an additional three months’ worth of learning than the gains made by district schools in these 12 years, that is the first finding. The second is that the gains are particularly large for 8th-grade African Americans. And they are refined, and the gains and charters are four times as large than the gains at district schools. The third major finding is that achievement gains are larger for students from low SES at charter schools than their counterparts at district schools, these are the three major findings.
Jason Bedrick: Right, okay. So before we get into analysis of the findings, you’re actually noted in a Wall Street Journal, congratulations on getting into the Wall Street Journal with Paul Peterson, that there was an exploratory precursor to these NAEP surveys in 2003, that showed that the average performance of charter school students in 4th-grade on reading in math was actually behind their district school peers. Your study shows that the gains that they make over time puts the charter school students ahead. So what happened between then and now that accounts for the difference in your finding?
Danish Shakeel: So, there was a study done in 2003, it was an exploratory study by NAEP and it found that the charters were underperforming district schools, and it gained a lot of media attention. If you look at the NAEP Explorer, which allows one to look at the basic differences between the two sectors, in 2005, the charters were underperforming the district schools, but if you look at 2017, the differences are not statistically significant. In some cases, the charters even go one point ahead than districts, so the achievement gap between the two sectors has closed. That itself gives us evidence of a higher rate of learning in charters over time. And the sector could have improved, it could have undergone selection issues, as you can say, but despite all those things, the controls don’t explain the differences in gains. So it could be that a charter school has become better in instruction as well.
Jason Bedrick: So what do you think actually it does account for these gains and why do we see that black and low income students seem to be benefiting so much more than the other groups?
Danish Shakeel: We don’t test for that, but maybe the kind of model that is new, no excuses model is working well for disadvantaged and black students. In a subsequent analysis, we are planning to include those hypothesis as well.
Jason Bedrick: Interesting. Can you speculate as to maybe why some groups are fairing so much better or gaining so much more than others?
Danish Shakeel: It could be because of mission orientation. And again, I’m not able to test this hypothesis, if you’re asking me to speculate.
Jason Bedrick: Yeah.
Danish Shakeel: So maybe some charters are driven with the mission that they want to serve these communities. And because of that drive, they focus on the learning of the underserved students. That could be the driving force, but again, I’m not able to test this hypothesis as yet.
Jason Bedrick: Right. And maybe for a future study too, it’d be good to break it down at the state level, to the extent that that’s possible to see if the charter sector in some states, because obviously charter laws differ dramatically from state to state and it would be really interesting to see if some states are posting more gains in the charter sector than in other states.
Danish Shakeel: Yeah, that’s a nice idea, but the NCES samples and also the NAEP samples do not collect the data for every state. So it is nationally representative, but once you go down to breaking for each state, then one cannot provide a full picture for every state. So we can consider that at least for this chat that charters are more prevalent and that could provide some better analysis.
Jason Bedrick: Certainly. Now, do you have any advice for policy makers based on what your study found?
Danish Shakeel: It seems to us that the opposition to charter is happening where the charters are actually performing very well. For example, in the Northeast, we see a large charter effect, but political opposition is also very large there. So it could be that the position to charter at the political level is happening where the charters are performing very well. The policymakers should be aware of that, they might use some data and their context to address that political backlash. And I definitely would like researchers to study charters in the context where the political affiliation is the largest. We are also applying to work on, in particular, testing this hypothesis that the political opposition is largest where the performance is largest.
Jason Bedrick: Our guest today has been Dr. Danish Shakeel, a post-doctoral research fellow at the Program on Education Policy and Governance at Harvard University’s Kennedy School of Government. Again, he is the co-author with Harvard’s Dr. Paul Peterson of the study, Changes in the Performance of Students in Charter and District Sectors of the U.S. Education: An Analysis of Nationwide Trends. Dany, thanks so much for joining the podcast.
Danish Shakeel: Thanks, Jason, for the pleasure to talk to you.
Jason Bedrick: Yeah, pleasure. This has been another edition of EdChoice Chats. If you have any ideas for authors you’d like us to interview for the Big Idea series, please send them to media@edchoice.org and be sure to subscribe to our podcast, follow us on social media @edchoice, and don’t forget to sign up for our emails on our website edchoice.org. Thank you, we’ll catch you next time.