Start Measuring Results

The Institute of Education Sciences should prioritize return on investment, student achievement, and statistically literate leadership.

A year ago, the National Association of Scholars (NAS) urged the Trump administration not to defund the Institute of Education Sciences (IES). It framed the benefit of the IES above all as a way to keep the states honest.

The great good done by the IES is to produce statistical information that is national in scope and that has been produced for enough years that it can provide substantial assistance to researchers who need comparable statistical data from different years. The IES’ statistical information is good in itself—and it also provides a good way to cross-check statistics provided by the states. States can manipulate their own measures of education proficiency—and when they do, IES’ statistics provide a way for states’ citizens to check about whether their state bureaucracies are providing honest statistics. America benefits when our different levels of government provide multiple statistical measures on educational attainment for the benefit of our citizens.

NAS did not claim that IES should not be reformed—and a good thing too. Amber Northern’s Reimagining The Institute of Education Sciences: A Strategy for Relevance and Renewal (with Adam Opp), which delves deep into the weeds of IES policy, makes clear that a host of intelligent bureaucratic reforms would do much to improve the way the IES operates. IES, on Northern’s account, is—to put it less politely than she does—sluggish, unfocused, and constrained by outmoded and counterproductive procedures, regulations, and legal restrictions. Dedicated education reformers would do well by IES and the country by engaging in chapter-and-verse administrative reform to enact what Northern recommends.

Northern’s Reimagining, however, is itself somewhat diffusely comprehensive. I would suggest several strategic priorities as a way to organize Northern’s generally worthy recommendations.

Above all, IES should make a priority of gathering statistics that allow policymakers and the public to judge the Return On Investment (ROI) of education expenditures, federal expenditures above all, then state and local expenditures. As NAS noted in Waste Land: The Education Department’s Profligacy, Mediocrity, and Radicalism, far too much education research either fails to measure the effectiveness of education programs at all, or satisfies itself with we spent money and there was improvement. The education statistics that Americans most need to know are those that will tell them precisely how effective each education dollar is, so that they may know how best to modify their education spending.

IES engages in a great variety of statistical information gathering, and not all of that data can or should be justified in terms of gathering ROI information. Yet as a rule of thumb, IES should make a priority of gathering information that maximizes ROI analysis. All components of IES should be encouraged to reconsider their missions within this framework. Indeed, IES might do well to provide a ROI evaluation of its own work, as it seeks to be most effective.

A second priority for IES should be to make a priority of gathering statistics that allow evaluations of educational achievement rather than statistics that evaluate educational “access.” The entire intellectual framework that puts “access” on a level with achievement distorts the conception of what education aims to do. Americans should want teachers to be as effective as possible within the classroom; getting students into the classroom is not the teacher’s job. Nor should it be the job of the Education Department, assuming it survives; if it is to be addressed at all at the federal level, that effort should be housed in the Department of Health and Human Services. Americans need statistics on, for example, the effectiveness of direct instruction compared with other pedagogies. IES should focus on how well students are served, not on how many are served.

IES ought finally to make a priority of educating statistically literate education professionals to work in state and local education administrations. Northern phrases the problem politely:

IES is statutorily required to ‘maintain research, evaluation, and statistics fellowships in institutions of higher education … that support graduate and postdoctoral study,’ (20 U.S.C. 9579). But graduate students who obtain doctoral degrees in education research are typically less likely to seek employment in state and local education agencies than graduate students who obtain master’s degrees in education. Moreover, numerous studies have found that master’s degrees in education themselves do not enhance the effectiveness of teachers, who often get these degrees for the pay bump associated with them. (p. 54)

A blunt translation: State and local education agencies are full of ed-school graduates who don’t know how to collect statistics, what they mean, or what they say about what better education policy should be. For heaven’s sake, get a thicker sprinkling of statistics wonks into the education bureaucracies, so all these statistics the IES gathers don’t go to waste because the policymakers learned about Freire instead of F Ratios. Northern frames her goal with equally polite caution:

Once placed in SEAs and large districts, they would be ambassadors for research and evaluation throughout their organizations. They would function as ‘translators and boundary spanners’ who understand and value both research and practice and can develop relationships in both camps. They would also act as intermediaries between IES and their own education employers, facilitating the flow of evidence-based practices from IES to school districts, nonprofits, and state agencies. (p. 55)

Education reformers should proceed beyond Northern’s recommendation, which pertains solely to the IES’s remit. They should aim to ensure that education leadership draws far more strongly from the statistically literate. Statistics wonks should be princes of their bureaucracies, not just ambassadors.

Most of Northern’s proposed reforms to IES are worth pursuing; the question is which to pursue first. I believe that ROI, achievement-not-access, and statistics-wonk education personnel would be three good priorities. IES would do well if it advanced on these three fronts.

Follow David Randall on X.

  1. In his well-reasoned essay, Randall argued that higher education should get more “statistics wonks into the education bureaucracies.” As always, the devil is in the details. Most statistics wonks are experts at dealing with randomness: random selection in surveys and random assignment in clinical trials. But most educational data is observational: the goal is observational causation and big issue is confounding. Most introductory statistics textbooks don’t even mention confounding. It’s like the elephant in the room. Many statistics wonks have not heard of Simpson’s paradox: the reversal of an observational association after taking something else into account. Of those familiar with this reversal, most consider it a curious anomaly. It is not. In an analysis of NAEP data, Terwilliger and Schield (2004) found over 100 instances of such a reversal in a single data set. Educational outcomes involve some of the most confounded statistics being measured today. To improve education substantively, we need statisticians who are statistically literate in dealing with observational causation and confouonding.

Leave a Reply

Your email address will not be published. Required fields are marked *