“Study Shows!” The Five Questions You Should Ask When Interpreting Research 

“Study shows how humans will live in inflatable moon villages!” 

Does it really though? 

In early October, EdChoice’s Dr. Michael McShane and the American Enterprise Institute’s Rick Hess armed attendees of EdChoice Academy with tools to analyze research — to sus out whether a study measures up to its headlines. 

This annual two-day, intensive crash course in Indianapolis prepares people new to education policy to defend school choice across the country.   

“What do you do if a colleague emails you an attachment and says, ‘Here’s this new study! I need to make something of it?’” McShane asked the room. 

“By asking five questions, you can do a pretty good job at analyzing what’s going on,” he said. 

What is Research? 

As the session got underway, attendees discussed how they have interacted with research at their jobs and what challenges have come up. 

“What is research?” Rick Hess interjected. 

A few moments of silence. Then a little laughter! 

Several definitions were thrown out — “a systemic way of finding answers,” “a way of quantifying information,” “the analysis of events.” 

McShane agreed, saying research is people “trying to better understand the world, come to knowledge, quantify things.” 

He cautioned that research is often used as a “sledgehammer, not a flashlight.” 

“We’ll start with a study, and I want you all to tell me what’s wrong with it,” McShane said. 

Question 1: Compared To What? 

The group looked at their first study: 400 eighth graders were fed broccoli every day for six months. Their height and weight were measured at the beginning, middle, and end of the study. On average they grew 1.5 inches and gained seven pounds.  

The conclusion is supposedly that “eating broccoli every day will help students grow.” 

“There’s no control group. Children are probably going to grow anyway,” one Academy attendee pointed out. 

Education research is particularly vulnerable to this kind of oversight, McShane said.  

For example, maybe you try out a new geometry curriculum, and when students know more geometry at the end of the school year, you claim it’s a success. 

“Well, I would expect kids studying geometry all year to know more at the end of the year,” McShane said. “Compared to what? How the district used to teach geometry? Other districts?” 

Question 2: Is The Sample Representative? 

“See every state’s favorite candy!” 

Academy attendees looked at a survey that assigned a favorite candy to each state based how much residents Googled that candy. 

“Here in Indiana we’re in M&Ms country. Milk Duds kind of own the South there, McShane said. “Anyone here from Kentucky? Swedish fish, didn’t see that one coming.” 

But Googling something doesn’t necessarily mean you like it. 

“‘Why do Swedish fish taste terrible?’ Turns out that would get you kicked in here,” McShane said. 

“If I was going to buy candy I wouldn’t go online first. I would just go to 7-11,” one attendee said. 

So we should ask, “is people googling candy types representative of people liking that candy?” 

Another example of the same problem is the famous diagram from World War II of a British Royal Air Force plane with bullet holes in it. 

The Royal Air Force wanted to add armor plating to their planes, so they mapped out where the planes had taken the most damage during bombings in Germany. The wings and body showed the most bullet holes, so they reasoned they should probably put the armor plating on the wings and body. 

But those were the planes that came back. 

The planes shot down over Germany most likely had more damage to the engines and cockpits. 

“It was actually a map of where it’s okay for planes to get shot,” McShane explained. 

So the question to ask here is, “Are these airplanes representative of the total sample of airplanes that went over to bomb Germany?” The answer is no. 

“That’s another huge thing that we see in education,” McShane said, noting that a study can show a new reading program worked in just one district, but supporters then claim the whole state or country should do it. 

“Where do you usually pilot a project? Usually places that opt in, right?” Hess said. “There’s a high degree of enthusiasm for it. Therefore you’ve got fidelity, you’ve got a lot of TLC, you’ve got resources. Then you say, oh it works, and you take it to a bunch of other places. What you are doing is you’re taking the specific technique to other places, but with none of the enabling conditions.” 

“What happens a lot of times is research winds up getting used as a way to not have to engage our common sense,” Hess said. “Where should we put plating on an airplane? Probably should put it where the pilot is and where the engines are. But you’ve got to be a special kind of weird research nerd to say no, let’s not pay attention to the obvious stuff. Let’s go count bullets instead.” 

“When you’re talking to real human beings about stuff related to schools and education, it’s very easy to get excited because you learned about a research study, and you think it’s persuasive. But there’s actually a really good reason normal human beings will shut out a lot of this, because they’re aware that they’ve been shopped a lot of garbage research over the years.” 

Question 3: Does The Writeup Match The Findings? 

A study that claimed kids have “three times too much homework” made huge splash in national news a few years ago. 

The study said first graders should have 10 minutes of homework, second graders 20 minutes, third graders 30 minutes, and so on, a very specific standard that could be problematic. 

It asked parents whether their kids had too much homework, collecting the answers from parents waiting in pediatrician offices in Providence, Rhode Island. 

“You probably had a lot of stressed out parents saying, we’re overwhelmed,” one attendee pointed out. 

“It’s turtles all the way down. It gets worse!” McShane said. 

It turns out the actual results of the study did not show three times too much homework in many grades, even by the survey’s specific standard. The study’s writeup — and the news coverage — did not match the findings. 

Question 4: “Is The Finding Meaningful?” 

Another study looked at CT scans on children’s heads, claiming that the risk of leukemia was three times greater in kids who received between five and 10 CT scans as a child. 

Here’s the problem: kids getting CT scans likely already have health problems. That’s why they’re getting the CT scan. Getting a CT scan itself doesn’t raise the chance of leukemia. 

“Just because the study finds a result, just because it passes the test of statistical significance, doesn’t actually mean it might alter anything we want to do in our lives,” McShane said. 

Question 5: Can These Findings Generalize? 

In another study, homeless people were given about $5,800. The results seemed promising. The homeless people obtained better shelter did not spend more on drugs and alcohol. 

However, they were screened for mental health and drug problems and had only been homeless for about six months. The vast majority of the homeless population would not meet this criteria, so the findings do not generalize very well. 

“The key is just to always be clear about what research actually says and what it doesn’t,” McShane said. “Over a long time and lots of studies and combining research with all these other things we care about, history and values, put all of that stuff together and it gives us a picture of how the world works and what are better courses of actions.” 

McShane also had a tip for how to avoid sacrificing clarity during TV hits. While sometimes the cost of speaking about a study for two minutes on TV is the nuance, he advised attendees to “find your own north star” and what their goals are in quickly breaking down the study. 

“You’ve got to make a choice. When you’re going on your TV hit, when you’re doing research, you can do the things that are going to generate more interest and get you invited back,” Hess said. “But over time, if you don’t take the easy bite, if you get a reputation for a guy who won’t cheap shot, those are the people who become seen as credible and authoritative.” 

Afterwards, EdChoice Academy attendees participated in small groups where they examined studies on their own and decided what they found and whether they were worth citing. 

If you missed this year, get in touch to come to our 2027 EdChoice Academy!