If U.S. News rankings were a cake, you wouldn't want to follow the recipe

UPDATE: the methodology described in this post is from the 2014 rankings. The 2015 use substantially the same methodology with some minor differences in application.

Imagine you want to bake a cake. Usually, you would want a recipe: the list of ingredients, the quantities of each, and the order and manner in which they are mingled.

Law school rankings are like that. But for the U.S. News & World Report rankings, the recipe does not match the cake you see. And that should disturb you.

Here are the items used in the methodology from 2014.

Quality assessment
Peer assessment score (0.25) [63% response rate]
Assessment score by lawyers/judges (0.15) [9% response rate, two-year average]

Selectivity
Median LSAT scores (0.125)
Median undergrad GPA (0.10)
Acceptance Rate (0.025)

Placement success
Employment rates at graduation (0.04) [variously weighing 22 of 35 different post-J.D. jobs and durations]
Employment rates nine months after graduation (0.14) [variously weighing 22 of 35 different post-J.D. jobs and durations]
Bar passage rate (0.02) [based on ratio of passage in jurisdiction]

Faculty resources
Average instruction, library, and supporting services expenditures per student (0.0975)
Financial aid expenditures per student (0.015)
Student-faculty ratio (0.03)
Library resources (0.0075)

The first problem is this: USNWR doesn't disclose all of the data it uses above. The second problem is this: USNWR discloses data it doesn't even use in its overall score.

If you opened up the magazine, then, the "recipe" would have a bunch of missing ingredients, and a bunch of useless ingredients.

You could figure out some of this data on you own, such as obtaining the ABA median LSAT and UGPA data. And you could understand why some of it is kept in the dark: for instance, keeping expenditures per student private to avoid an arms race, or to minimize the secret formula for weighing employment rates to avoid schools gaming the employment statistics as they had for years. Some try to reverse-engineer the scores to reveal the data that's otherwise hidden.

But it's notable that the final score--and final ordinal rank--has a majority of its data concealed from the public eye. Indeed, its glossy print edition includes six columns of data that the magazine itself deems irrelevant to its overall score. It's one reason why professors like Brian Leiter constantly implore commentators to focus on the data, not the ranking--and it should call into question why so much data is concealed.

Tomorrow, U.S. News & World Report will disclose its ordinal rankings, likely with a slightly modified methodology. But it's important to recognize that this ranking--this cake--was baked with ingredients we don't have the pleasure of seeing.