In the current New Yorker, Malcolm Gladwell entertainingly explains why computing a unidimensional ranking of educational quality from multidimensional indicators is a fool’s errand. In the case he examines, the project is to identify the best schools in order of quality, when the best school does not exist any more than a best kind of music exists. The absurdity of the project appears when we find out how much the outcome varies with the weights given the factors and realize that sensible people will weigh them differently. The pointlessness of the project deepens when we learn that the numbers being used to indicate quality are themselves of dubious value and subject to misreporting. In the first iteration of my Ranking Game, I showed that a law school’s rank might change substantially if the ranker includes faculty/student ratio (on which higher is better) instead of the opposite of (i.e., negative of) student/faculty ratio (again, on which higher is better). Which one is the better measure of school quality, faculty/student ratio or student/faculty ratio? Gladwell and Donald Downs rightly point out that price ought to be an important component of any rational decision. But no ranking designed to reach a heterogeneous audience will include school price as a fixed factor. If it were included, many readers would recognize that the ranking does not fit their situation because they qualify for a scholarship or because they do not attach the same importance to price. Conversely, leaving price out of the equation helps to obscure the fact that the ranking fits few prospective students. Omitting price tailors the ranking to those who have the means to pay any toll, which supports the observation that the priorities embedded in US News’s ranking are more consistent with aristocratic values than the values of access and efficiency. The omission of price might mislead students into taking on substantial debt in order to buy a negligible improvement in education. Even if the numbers used were good indicators and appropriately weighted, they would assume undeserved importance in the decision making process. Because numbers give a false sense of scientific precision, it is hard for a decision maker to override them with fuzzy or soft non-numerical facts that seem less reliable. As Paul Simon sang, “When times are mysterious, serious numbers will speak to us always. That is why a man with numbers can put your mind at ease.” Speaking of numbers that speak to us, Michael Alexeev and I have made an empirical study of Gladwell’s conjecture that rankings are a self-fulfilling prophecy. In an unpublished draft available at SSRN.com, we present evidence for what I call the “echo effect.” US News publishes its ranking in the spring and then collects reputation data in the fall. It should be no surprise that the reputation data they gather reflect the ranks they published a few months earlier. Analysis of their numbers confirms that they are hearing echoes of their own pronouncements. Rankings misguide readers in other ways, too. By converting cardinal numbers to ordinal, a ranking wipes out differences in degree. Tiny differences often loom large. Using US News’s dubious measure of law school quality, equal size steps are made from rank 1 to rank 5, 5 to 15, 15 to 27, 27 to 54, and 54 to somewhere above 102. If a person could rationally choose Chicago over Yale, a person could equally rationally choose a law school ranked 102 over one ranked 54. But that is not the message conveyed by the ranking. To most readers, there appears to be a far greater difference between number 108 and 54 than between 1 and 5. Gladwell and Downs do not mention another dimension of evil that inheres in rankings: bad incentives for the entities being ranked. To take one example, law schools have changed their admissions policies to improve their rank in US News. Some schools have reduced the size of the entering class to raise their LSAT median. Many schools admit students with high grades garnered in easy curricula over students who achieved lower grades in difficult curricula. And the incentives cascade. A rational response to the admissions policies of desirable schools would appear to be for students to avoid taking challenging courses that might put their “A” average in jeopardy. Rankings mislead many, in many ways. Jeffrey Stake Indiana University Maurer School of Law Bloomington, Indiana
