Malcolm Gladwell has written his share of interesting and penetrating essays in The New Yorker in recent years. He has also authored such best-selling books as Blink, which is about rapid cognition and intuition, and The Tipping Point, which addresses the factors that contribute to unexpected change. The relevance of Tipping Point has received another big boost by the recent happenings in Egypt. Among Gladwell’s attributes is his ability to question and challenge conventional wisdom.
The virtues of Gladwell’s scalpel are on display in his New Yorker essay (February 14 and 21 issue) attacking U.S. News and World Report’s famous (or notorious) national “Best Colleges” ranking guide. Even though U.S. News is now defunct, the Guide survives and is used by millions of families. “The rankings have taken on a life of their own,” as Gladwell writes. Given the difficulty and complexity—often the sheer mystery—of knowing how schools compare, the Guide’s assignment of numerical rankings appears to have been a blessing, as it simplifies the task of evaluation for millions of students and parents. But what if it amounts to a false promise?
The Guide has been questioned by some empirical researchers, including Michael Bastedo of the University of Michigan and Jeffrey Stake of Indiana University, as well as by schools that feel unjustly slighted by its determinations. But seldom has it found itself in the sights of a national magazine like the New Yorker. Gladwell’s critique provides convincing evidence that consumers should take the Guide with a big spoon of salt.
The heart of the problem lies in the use and abuse of measurement. Gladwell tellingly begins his piece by comparing the Guide’s logic and methodology to Car and Driver’s recent comparison test of three sports cars: Chevy’s Corvette, the Porsche Cayman S, and the Lotus Evora. (Porsche won, followed by Corvette and Lotus). Car and Driver’s report is unreliable, Gladwell avers, because it applies the same twenty-one criteria to sports cars that it applies to all vehicles, thereby ignoring special concerns that sports car buyers have, such as the way the car looks. Nor did the test give much weight to cost, which matters a lot to consumers. Car and Driver attempts to have its cake and eat it, too, but “it’s an act of real audacity when a ranking system tries to be both comprehensive and heterogeneous at the same time.”
Unfortunately, that is precisely what the U.S. News Guide does by relying on seven weighted variables to rank schools on the same list that differ as much as Ohio State and Reed College: undergraduate academic reputation; graduation and freshman retention rates; faculty resources; student selectivity; financial resources; graduation rate performance; and alumni giving. No attempt is made to keep the comparison of apples to apples, and prunes to prunes. For example, the University of Illinois ranks one slot higher than Yeshiva University on the list, though two institutions could hardly differ more in size, character, purpose, and geography. The problem is compounded by the complexity and subtlety of the subject matter. It is one thing to rank inanimate objects like a car, yet another to assign a number to a huge multiversity replete with dozens of schools and departments and over forty thousand students. As Gladwell observes so devastatingly, “How on earth can anyone propose to assign a number to something” as complicated as this?
Other problems stand out in the Guide’s methodology. As in Car and Driver’s rankings, price is not given much weight. Given the growing crisis of student (and parental) debt that numerous sources have chronicled (including a CNBC documentary and in a new book by Andrew Hacker and Claudia Dreifus, (Higher Education? How Colleges Are Wasting Our Money and Failing Our Kids—and What We Can Do About It) this omission is significant. Never before has the price of education been more relevant to families, yet it receives scant attention in the nation’s primary source of higher education rankings.
The subjective and mysterious way the Guide’s six-person team determines scores for the seven variables it considers important exacerbates the problem. For example, the team uses six factors to measure student “engagement,” which is a component of “faculty resources.” Such factors include faculty pay. But faculty pay is often a function of research accomplishments, which—as many critics of higher education claim—is not always associated with good teaching. Indeed, the opposite can be true. Another example is the notoriously subjective “reputation score,” which can boil down to conventional wisdom derived from the Guide’s determinations. According to Gladwell, “The U.S News ratings are a self-fulfilling prophecy.”
In defending the Guide team’s decision to downplay the cost of education, team leader Robert Morse told Gladwell that “We’re just saying we’ve made this judgment. We’re saying we we’ve interviewed a lot of experts, we’ve developed these academic indicators, and we think, these measures measure quality schools.” Gladwell replies that “As answers go, that’s up there with the parental ‘Because I said so.'”
Promising Impossible Precision
In some respects, the Guide’s conclusion remind me of Supreme Court Justice Potter Stewart’s famous statement about obscenity: “I shall not today attempt further to define the kinds of material I understand to be embraced . . . [b]ut I know it when I see it.” A very subjective position, but also not entirely bereft of meaning. Much of the evidence we act upon in life is gray, and we accept it as such; and most families would rather have their children attend Harvard (setting aside price for the moment) than an obscure satellite state school. The problem with the U.S. News Guide is that it practices what Alfred North Whitehead called “misplaced concreteness.” It promises precision when precision is impossible. After all, Stewart did not indulge in assigning numbers to the variety of pornographic depictions he witnessed as a justice.
By relying so much on faculty and administrative input, the Guide reflects the entrenched status quo. But Gladwell’s critique is especially interesting because it takes place at a time when more and more people are questioning the establishment. Hacker and Dreifus critique many costly and cumbersome institutions that have become progressively less interested in teaching, while such writers as Walter Russell Mead depict domains of higher education as a medieval-type “guild” that is often more interested in protecting its own interests and prerogatives than those of the clients and public it serves.
College guides can be useful. When my children were looking into college in the 1990s, one book covering the “Best Buys” in higher education was very useful. It considered some of the factors that the U.S. News Guide treats; but it also dealt with bang for the buck, presented fairly in-depth case studies of each campus it covered, and provided comments from parents and students who knew about the school under discussion. Loren Pope’s Colleges that Change Lives has helped many families by offering in-depth, on the ground portrayals of real college life. Other useful guides consider the status of free speech, intellectual diversity, and political correctness—important factors the U.S. News Guide apparently considers immaterial.
In reading Gladwell’s critique, I was reminded of the argument Stephen Jay Gould made in his classic book, The Mismeasure of Man (1981). Gould took aim at various theories of biological determinism and at the idea that human worth can be reduced to measurements of intelligence. Two targets of Gould’s critique are undue “reification” (our inclination to make abstraction concrete entities) and “ranking,” which he described as the “propensity for ordering complex variation as a gradual ascending scale.”
The U.S. News Guide epitomizes these two fallacies. It takes the complex tapestry of student need and institutional variation and turns them into a single ordinal ranking. The U.S. News ranking of departments for graduate training is less objectionable, for such rankings are predicated on the research productivity of faculty, which is comparatively more ascertainable than the overall quality of institutions for undergraduate teaching and education (though care and caution are in order in both areas). But the quality of departmental research is often a poor proxy for the quality of undergraduate education.
The national obsession with the U.S. News rankings perhaps reflects cultural penchants or assumptions that merit some reconsideration. One, as Lynn Payer depicts in Medicine and Culture, is our proclivity to rely on ostensibly objective scientific criteria in cases in which judgment and uncertainty are more appropriate. (Some matters are amenable to scientific precision, others are not.) Another is the fetish that we make of the status of one’s undergraduate institution. One of our distinguishing characteristics as a liberal democracy is our lack of an aristocratic past. But we compensate for this lack by placing undue social and moral significance on where one goes to school.
Gladwell’s essay is a reminder that we should view higher education through a more rational lens.
