The Achilles Heel of the U.S. News Rankings

In 1983 U.S. News & World Report came up with what Ben Wildavsky, a former education editor at the magazine, described as “a journalistic parlor game.” The magazine had just conducted a successful survey of U.S. leaders to identify the most influential Americans. Why not, the editors asked, use a similar approach to identify the country’s top colleges and universities?

So U.S. News sent surveys to college presidents around the country asking them to pick ten colleges that provided the best undergraduate education in their particular academic niche. The magazine published the results in 1983 and again in 1985, and by 1987 the project had morphed into a free-standing guidebook entitled America’s Best Colleges. “No one imagined that the rankings would become what some consider the 800-pound gorilla of American higher education,” recalled the late Alvin Sanoff, the longtime managing editor of the rankings project.

The gorilla continues to stalk U.S. higher education. Last week Daniel de Vise of the Washington Post reported that “a small but determined” group of college presidents in the Washington-Baltimore area is now boycotting the “peer assessments” questionnaire that U.S. News & World Report sends them every year as part of its process of updating its college rankings. Their protest follows a report last year that another group of college presidents across the country had pledged to do likewise.

It’s easy to understand why college presidents don’t like U.S. News butting into their affairs in the first place and might be inclined not to cooperate at all (as Reed College has done). John Burness, the former communications chief at Duke University, probably spoke for most of higher education when he observed in 2008 that the precision that U.S. News ascribes to its rankings “is, on the face of it, rather silly.”

But the rankings are not going to go away, as least as long as they continue to sell magazines, and few colleges and universities have the stomach to stop cooperating entirely. So the focus of debate seems to have shifted to one of the more dubious elements of the U.S. News formula: the “reputational” survey. To be sure, this part of the formula has been expanded to include not only presidents but provosts and others who would bring a (perhaps slightly) broader perspective. But presidents are still the key.

The rebellious presidents quoted by the Washington Post argue convincingly that the reputational survey is a sham because they simply do not know all that much about the competing institutions on which they are asked to comment. As Sanford Ungar, a former journalist who presides over Goucher College, told de Vies, “I would be better able to fill out a survey on refrigerators than on colleges I’ve never visited, never interacted with.”

I tend to sympathize with their view. Every year I find myself moderating a panel of mostly college presidents charged with advising high school students about how to approach applying to college. I’ve always had my doubts about the usefulness of this exercise, because it strikes me that college presidents, preoccupied as they are with the fortunes and advantages of their own particular institutions, are among the least well positioned of their professional colleagues to offer sound and objective advice to nervous adolescents on how to navigate the admissions waters. And what they know about schools with which they compete probably does not go much beyond how their endowments are faring.

Something else that might be in the back of presidents’ minds might be flood of promotional materials that they receive from peers seeking to bolster their institutions’ images. College presidents report that they are regularly on the receiving end of glossy brochures with folksy covering letters from fellow presidents starting out along the lines of “I just thought that you might be interested in reading some of my recent speeches and learning how well our faculty is doing on the publication front.” Most presidents I have spoken with find them more amusing than serious. But the senders seem to be doing it for a reason, however desperate it may be.

Then, of course, there is outright gaming of the system. The most egregious example of the latter came to light last June when the former director of the Office of Institutional Research at Clemson University let it be known that senior officials at that school had deliberately given low reputational scores to rival universities in order to bolster Clemson’s ranking. (The university denied the charge.)

The reputational rankings are also open to methodological challenge. To reduce strategic voting, the magazine eliminates the two highest and two lowest scores of each college, just as the Olympics started to do in response to biased judges from the USSR and East Germany. Last spring, Synovate, an opinion-research firm outside Chicago, collected the reputational surveys sent to 4,273 people. Only 48 percent responded, and that included an unspecified number of non-responsive “responders” who returned the questionnaire but said they didn’t know enough about other schools to judge them fairly.

Perhaps the most distressing thing about the reputational surveys is that they tend to become self-fulfilling prophecies. Nicholas A. Bowman and Michael N. Bastedo, two academics who have studied the effects of reputational surveys, talk of the “anchoring effect” under which people asked to make judgments in ambiguous circumstances start with whatever information they already have at their disposal and then go on from there. In the case of reputational surveys, what college presidents know at the outset is last year’s U.S. News rankings. So in a paper prepared for the recent meeting of the American Education Research Association they come to the conclusion that “rankings drive reputation, and not the other way around.”

In his fine new book on the globalization of higher education, The Great Brain Race: How Global Universities Are Reshaping the World (Princeton), Wildavsky describes how rankings are becoming increasingly important around the world as countries trying to create “world class” universities look for ways to calculate where their institutions stand in the global pecking order. This means that we are going to be hearing a lot more about the strengths and weaknesses of reputational on the world stage. If Sanford Unger does not feel confident making judgments about Goucher’s U.S. competitors, how is he going to feel when the London Times asks him about Tsinghua University in Beijing?