Everything People Care About More Than Standardized Test Scores

There’s an old story involving a woman finding her husband wandering around outside, eyes fixated on the ground. “I’ve lost my ring,” he explains. “Where did you lose it?” she asks. “Inside the house,” he says. Perplexed, she asks, “Why are you looking for it out here?” “It’s too dark inside,” he replies. “There is much more light out here.”

This tale depicts observational bias, in which humans are disproportionately attracted to obvious information at the expense of other, potentially more accurate or truthful information. In social sciences, this bias has been discussed as a warning since at least the 1960s: Scrutinizing the easily measurable sometimes prevents actual discovery.

Education research is as susceptible to this danger as any other social science. We are inundated with an unfathomable amount of data, so we expect to be able to know everything we want. Intuitively, we surely have the data we need to answer the most basic question the public has about its schools: are they working?

It’s not a simple question to answer, of course. For one, people have different goals and priorities for K-12 education. Some values are pretty ubiquitous, such as literacy. But there are varying degrees to which people value their schools helping their students prepare for advanced education, get ready for the job market, become well-adjusted citizens, carry on cultural traditions, and so on.

But even if we identified a consensus around educational goals—across geographies, classes, and cultures—we still have a knowledge problem. With a hundred thousand schools in the US, each with unique students, teachers, and administrators, how do you compare apples to apples?

The best available answer has been standardized test scores. States administer uniform tests to students throughout their jurisdiction, and for decades the federal Department of Education also has administered uniform tests to compare students between states.

That leads me to some recent survey data we’ve collected at EdChoice. We asked parents to identify up to three out of a possible six indicators they look at to know if their child’s school year has been successful. The figure below indicates how the results shook out:

Of the indicators presented, standardized test scores were a clear least favorite. The most favored options were relationally oriented, focused on personal feedback and not simply numbers. Notably, if you look at the crosstabs, parents’ education level does not really make them more or less likely to value the test-based indicators, but higher levels of parental education were indicators for greater preference for communication-based feedback.

Any market economy requires information to work effectively. Customers have to know what they are actually receiving in a transaction in order for markets to “clear,” as economists would say. As the K-12 education space becomes more market oriented as educational options become more accessible, parents will naturally expect schools to earn their confidence. From a parental accountability perspective, standardized test scores are not a favorable way to receive that information.

Parents, it seems, would rather develop relationships with their children and their children’s teachers to gain better understanding of what is actually happening in the classroom. They would rather know their child has a good attitude toward school and enjoys learning. Numbers are helpful, and plenty of parents told us they look at report cards and low stakes tests the school administers. But the major, standardized tests designed to help governments and researchers do their jobs better? The appeal is not as widespread.

Recent research suggests that isn’t really a problem. Of course, the whole premise behind using testing as an accountability measure assumes that success on a test is an effective indicator of long-term outcomes like educational attainment and job prospects. And there’s evidence suggesting that isn’t always the case, at least in school choice programs.

Most recently, a team studying the largest voucher program in Ohio found positive educational attainment effects despite disappointing results for state test scores. The authors specifically mentioned that the disconnect between state test scores and long-run outcomes suggests the state test is an ineffective tool to evaluate schools participating in the program (p. 16). Ohio private schools found ways to provide particular educational value without the specter of tests overshadowing their curricula. It doesn’t mean tests are inherently or ubiquitously useless, but it does indicate that even the best attempts at objective apples to apples comparisons can only be generalized so far.

Families don’t need the same kind of robust research that policy analysts do. Families are not making decisions about how K-12 education in the United States should be. They are making decisions about their child’s education. When your context is that specific, qualitative, relational feedback could be more useful.

Don’t get me wrong, it’s still very valuable to see the big picture through data. But prioritizing family-based accountability over more technocratic solutions is not rejecting the need for information and feedback altogether; instead, it properly values the kinds of information most important to those closest to the students.

This was originally published to our Substack.