Test Scores Matter—But Not How You Think

Why test-based accountability may miss the bigger picture in evaluating private schools in choice programs

A recent article in Education Next, “The Predictive Power of Standardized Tests,” highlights the strong relationship between middle school test scores and long-run educational outcomes such as college attendance, degree attainment, and earnings. Using data from Missouri public schools, researchers Darrin DeChane, Takako Nomi, and Mike Podgursky found that test scores in subjects like math and English predict high school GPA, college enrollment, and degree attainment.

This is important research, especially as debates over the value of standardized tests continue to swirl. The authors rightly argue that test scores offer a timely and useful signal—one that can help parents, educators, and policymakers identify which students may need additional support.

Now, if you’re a parent (like I am), that might make you pause. And if you’re a school choice supporter (like I am), it might make you squirm.

Why? Because several studies of private school choice programs have found something different: students in choice program sometimes score lower on standardized tests, but end up doing better in the long run. Many high-quality evaluations of private school choice, such as those in Washington, D.C., Louisiana, Indiana, and Milwaukee, have found a disconnect: lower test scores, but better long-term outcomes like high school graduation and college attendance, persistence, and degree attainment.

Taken together, these findings raise questions that strike at the heart of school accountability and choice policy.

A Test Score Paradox

This isn’t a new debate. A 2018 report from the American Enterprise Institute tackled this question head-on. Collin Hitt, Mike McShane, and Patrick Wolf reviewed multiple studies of school choice programs and found a puzzling disconnect: test scores didn’t always align with broader measures of success. In some cases, students in voucher programs scored lower on standardized tests than comparable students in public schools, but they were more likely to persist in school and go to college.

This test score–attainment gap has become one of the most interesting—and misunderstood—puzzles in the school choice research world. And it’s led some to ask whether we’re over-relying on test scores, especially when evaluating the success of private schools in choice programs.

Prediction Is Not the Same as Causation

One important caveat often missed in reactions to the Education Next article: just because early test scores predict later success doesn’t mean that raising test scores necessarily causes greater success.

Think of it like a credit score. People with high credit scores are more likely to get loans, buy homes, and build wealth. But raising someone’s credit score overnight—say, by gaming the formula (to be clear, I’m not suggesting anyone should do this!)—won’t automatically make them financially secure. The score is just a proxy for deeper patterns (underlying behavior, skills, conditions, traits, etc.).

The same logic applies to test scores. The Education Next study shows that students with higher test scores in middle school tend to earn higher GPAs, attend college, and earn more as adults. But that’s not the same as saying: “if we raise a student’s test score this year, we’ll boost their future earnings.”

That’s the leap many policymakers and commentators make—and it’s where things get risky.

What we’re really interested in, especially in school choice research, is trajectory change. Did a school, a teacher, or an opportunity actually change a student’s life? Did it alter the arc they were on? In econometric terms, we care about treatment effects, not just correlations.

This is where private school choice studies are particularly revealing. In some cases, students move to a private school, score the same or even lower on standardized tests—but later graduate at higher rates or are more likely to attend college. That’s a sign that something important happened outside the narrow slice measured by the state test.

Apples to Apples: Public School Students Only

It’s also worth highlighting a couple key details about the study. First, the CALDER paper emphasizes odds ratios, a statistical tool that compares how much more likely one group is to reach a certain outcome relative to another. For instance, the authors found that an 8th grader scoring Proficient in math was more than 13 times as likely to earn a postsecondary degree as a peer who scored Below Basic. In English Language Arts, the gap was even wider.

Second, the authors make clear that their analysis compares apples to apples: all the students in the study were in Missouri public schools in 8th grade. That’s an important nuance. When students move to a private school, particularly one with a different curriculum not aligned to state tests, their test scores might dip initially, not because they are learning less, but because they are learning differently. Over time, you’d likely find the same patterns: students who score higher go on to achieve more. But because the state assessments are tailored to the public system, they may not immediately reflect what’s happening. That’s why long-term outcomes matter more than short-term test scores—especially when evaluating schools across different sectors.

Why State Test Scores Might Matter More for Public Schools

Let’s talk about something that rarely gets said out loud: state test scores probably matter more—both practically and politically—for public schools than they do for private ones.

Why? Because of how the systems are built.

Public schools are directly accountable to the state. The curriculum is aligned to state standards. The tests are aligned to those standards. Teachers are trained to teach to them. School report cards, state interventions, and in a few states even funding formulas often hinge on student performance on those exact tests.

In short, test scores are deeply embedded in the machinery of public education. They aren’t just a measure of student learning—they’re a tool of system-level accountability.

That’s not necessarily a bad thing. In a large, centrally governed system like public education, state tests can provide a consistent (if imperfect) yardstick. They can help identify struggling schools. They can give policymakers a snapshot of equity gaps. And they matter for districts that have little other way to demonstrate progress to the public.

But now imagine a private school participating in a voucher or ESA program. It might use a very different curriculum, such as classical, religious, or Montessori. It may have different sequencing. It may intentionally avoid teaching to the test. And the parents who choose it might care more about school culture, discipline, religious formation, or a safe environment than a state math test score.

So when students at these schools take a test built for the public school system, it’s not surprising that the results might be lower—at least at first.

That doesn’t mean the school isn’t working. It means it’s doing something different.

What This Means for School Choice

The Education Next article reminds us that test scores can matter a lot, especially in large public systems where other forms of information are hard to come by. But in the world of private school choice, they may not tell the whole story.

A school that boosts confidence, builds strong character, or re-engages a disengaged student might not generate big test score gains right away. But it might change the trajectory of that student’s life—leading to graduation, college, or a better job down the line.

And that’s exactly what the AEI report showed. In some cases, the schools that seemed “bad” on test scores actually produced better long-term outcomes.

So for policymakers and researchers, the message is this: test scores are useful, but not sufficient. In public systems, they’re often the best tool we have. But in choice programs, especially those serving disadvantaged families, we should be cautious about using test scores as the sole—or even primary—measure of quality. Context matters, and in private choice programs, testing offers value when they’re used to generate information for parents to make informed decisions rather than as a bludgeon to punish certain schools.

Families don’t choose schools based on test scores alone. They choose schools that align with their values, that make their kids feel safe, or that treat them as individuals. That’s the whole point of school choice: to give families the freedom to pursue what matters most to them.

So yes, test scores matter—but not how you think. And not always where we’re looking.

This was originally published to our Substack.