The Tennessee pre-K study doesn’t tell the whole story

In January, a team of researchers at Vanderbilt University released a study that seemed to be a serious setback to the push for nationwide, universal pre-K programs. The study found that children who enrolled in a Tennessee pre-kindergarten program in 2009 and 2010 had worse test scores and behavioral outcomes as sixth graders than children who didn’t. The study was cast by pre-K critics as another blow to President Biden’s struggling Build Back Better bill, but the ramifications were even bigger. Media outlets and pundits wondered: Is pre-K actually bad?

The Tennessee study was carefully designed, comparing almost 3,000 low-income children who were randomly chosen from a group of applicants to a similar control group who were not chosen. But pre-K is not bad, and the problem is not the study. It’s how the language and techniques of academic research are mistranslated into how education policy is understood by the public and policymakers alike.

Pre-K has been offered in various states and municipalities for decades, producing scores of academic studies. Most have found positive effects on children. Less than a week after the Tennessee study was released, new research from Indiana found positive results for pre-K on test scores in grades three and four. As in Tennessee, the program serves low-income families. We’ve all learned to stay focused on polling averages in tracking political contests because even well-designed polls will sometimes yield inaccurate results. Research findings should be handled the same way, and the research equivalent of the polling average for pre-K — looking at multiple studies rather than just one — remains consistent and strong.

Still, negative results demand attention. One explanation for the poor outcomes is that pre-K education in Tennessee circa 2009 and 2010 wasn’t as good as it should have been. While child care is, all by itself, an important benefit for working parents, it’s not enough, academically, to just open a room for little kids to be in all day. A good pre-K classroom has well-trained teachers who understand how to structure the environment to encourage the development of language and cognitive skills. This doesn’t mean rigid instruction, but rather a lot of well-designed opportunities for enrichment and play.

The authors of the study offer evidence that Tennessee’s program was comparable in quality to other states. But there’s reason to believe the overall quality still wasn’t so good. Between 2009 and 2012, researchers, including two of the new study’s co-authors, evaluated a sample of 160 Tennessee pre-K classrooms with a widely used survey instrument called the Early Childhood Environment Rating Scale (ECERS), which evaluates classroom design, environment, curricula, discipline, and the strategies teachers use to promote language and literacy. Only 15 percent of the classrooms scored as “good” or above. Eleven classrooms scored below “minimal” quality.

The Tennessee legislature went on to pass the Pre-K Quality Act of 2016, which was designed to improve classroom curricula, provide training for pre-K teachers, and strengthen coordination with elementary schools.

While the negative results of the Tennessee study were very unusual, the finding of diminishing returns was not. A number of other studies have found that the academic benefits of pre-K sometimes fade over time. The authors speculate whether some pre-K classrooms might have been too focused on discrete, measurable goals like “knowing your ABCs” at the expense of broader literacy and executive functioning skills that matter in later years. It’s a fair question.

But the Tennessee study and resulting blowback also illustrate a broader problem, not limited to education, in how research methods define complex systems and how the media describes those results.

Research on what works is important, but it has its limits

The negative effects of pre-K in the Tennessee study were “statistically significant.” In normal language, “significant” means “substantial” or “non-trivial.” In statistics, “significant” means something else: “A difference that is very probably not random.”

Imagine dumping a quart of white paint onto a football field. Statistical significance means “the field is, on the whole, unambiguously less green and more white than it was before you dumped the paint.” It does not mean “a substantial part of the field is now white.” An effect can be statistically significant and practically insignificant at the same time.

The Tennessee study found that children who attended pre-K had a 97.1 percent attendance rate in the sixth grade, while children who did not attend pre-K had a 97.5 percent attendance rate; there were no significant attendance differences in grades 1-5. This finding was reported in New York magazine as “pre-K attendees were also significantly more likely to miss class.” Which they were, in statistical terms — but what it ultimately refers to is a difference of 0.4 percentage points in one year out of six. (The same observation can be made about some of the much larger number of positive pre-K results: They are statistically significant but not especially large.)

The much bigger debate about whether pre-K is worth the kind of huge national investment proposed by President Biden is often ill-served by the vocabulary and practice of modern social science, particularly the kind that leads to journal articles and tenure promotions.

Evan Vucci/AP

The Tennessee study uses powerful statistical techniques designed to find meaning in a fog of information. In a world awash in personal narrative, anecdata, and ideology, these methods, which have been greatly refined and improved over time, are incredibly important. They help distinguish causation from correlation, pattern from chance, truth from fiction.

But they also impose a very specific mental model on everything they examine. Studies are designed to lock onto a discrete action and determine what subsequently happened as a result of that event, and that event alone. They are perfect for evaluating something with which we are all now too familiar: vaccines. FDA trials randomly assign people to one of two groups. One gets the medicine, the other a placebo. They wait for a certain amount of time and see if the people who got the medicine are less sick. Not coincidentally, the authors of the Tennessee study describe pre-K as a “treatment,” standard language in social science.

The problem is, pre-K is not very much like a vaccine. Educating a child is more like building a house. Nobody thinks of walls, windows, and roofs as discrete interventions designed to keep people warm and dry. They are components of a larger whole. If the roof leaks, you get wet. If the windows break, you get wet. Foundation cracked? Wet. All the pieces have to work together at the same time.

Many early education initiatives, like Head Start and the Tennessee program, have been provided to children living in impoverished, sometimes traumatic environments. The public schools in their neighborhoods are often underfunded and poorly performing. Jobs and health care are scarce. Giving them pre-K can be like helping an unhoused person by building a single wall on a vacant lot. One wall is better than no walls, but they are still exposed to the elements above and on three sides.

Some of the most effective early-learning programs provided a range of social, parent, and health supports beyond education. (The Biden plan, which includes funding for early child care, a child tax credit, improved health care coverage, community college, and so on, takes a similar path.) Some of the least effective were implicitly premised on the hope that an extra year of school could inoculate children from the risk of academic failure, saving policymakers the trouble and expense of improving the next 13 grades. If the consistent initial benefits of early education sometimes fade, we should focus on the schools and grades where gains diminish.

The distinction between components and structure helps explain a longstanding conundrum in education research. At the level of nations, populations, and individuals, the benefits of education are enormous. Highly educated people do better on nearly every economic and social measure: earnings, health, longevity, and so on.

But researchers have thus far had trouble isolating the effects of specific parts of the educational whole. It’s absolutely impossible to write an article like “Effects of a Statewide Pre-Kindergarten Program on Children’s Achievement and Behavior Through Sixth Grade” and publish it in a peer-reviewed academic journal without at least 20 years of formal education. But the statistical techniques you pick up along the way aren’t yet up to the task of explaining exactly why.

The pre-K debate is also subject to some bigger misconceptions. Noah Smith, an economist and popular blogger, offered a fair and thorough summary of the pre-K research in his Substack newsletter, concluding that, while pre-K may provide more benefits to disadvantaged children who lack an enriching, stable home environment, “there are lots of kids who will probably be hurt by forcing them into universal pre-K programs.” But no universal pre-K programs are mandatory; in the vast majority of states, even kindergarten isn’t mandatory. In Tennessee, only 22 percent of the state’s 4-year-olds are enrolled.

There’s an important place for research like the Tennessee study in conducting education policy. It can help educators understand what works best and how to improve.

But for more existential questions — like whether universal pre-K should exist in the US at all — it’s useful to start with what privileged people give to their own children. The Capitol Hill neighborhood in Washington, DC, for example, is home to many of the highly educated staffers, lobbyists, and lawmakers who will help determine the ultimate fate of Build Back Better. Where are their 3- and 4-year-old children educated? Many are in pre-K at tuition-free public schools. Sometimes counterintuitive research findings are that way for a reason.