Did "boycotting" the USNWR law rankings affect those schools' peer scores?
On the heels of examining the peer score declines of Yale and Harvard in this year’s USNWR rankings, I wanted to look at the peer scores more generally.
Earlier this year, when I was modeling USNWR law rankings, I offered this thought:
I used last year’s peer and lawyer/judge scores, given how similar they tend to be over the years, but with one wrinkle. On the peer scores, I reduced any publicly “boycotting” schools’ peer score by 0.1. I assume that the refusal to submit peer reputational surveys from the home institution (or, perhaps, the refusal of USNWR to count those surveys) puts the school at a mild disadvantage on this metric. I do not know that it means 0.1 less for every school (and there are other variables every year, of course). I just made it an assumption for the models (which of course may well be wrong!). Last year, 69% of survey recipients responded, so among ~500 respondents, the loss of around 1% of respondents, even if quite favorable to the responding institution, would typically not alter the survey average. But as more respondents remove themselves (at least 14% have suggested publicly they will, with others perhaps privately doing so), each respondent’s importance increases. It’s not clear how USNWR will handle the reduced response rate. This adds just enough volatility, in my judgment, to justify the small downgrade.
Was that true?
USNWR’s methodology provides that it withdrew the survey responses of “boycotting” schools: “Peer assessment ratings were only used when submitted by law schools that also submitted their statistical surveys. This means the schools that declined to provide statistical information to U.S. News and its readers had their academic peer ratings programmatically discarded before any computations were made.” So my first assumption was right.
But did it affect those schools adversely?
But among these ~60 schools, 7 of them saw an increase in their peer score (12%). Another 28, nearly half, saw a decline. The average effect on their peer score was a decline of 0.043, slightly less than than the 0.1 I projected, but still an average decline.
Another 130 or so did not boycott. 29 of them (22%) saw an increase in their peer score, and 36 (27%) saw a decline—a mixed bag, with declining schools slightly outpacing increasing schools. The average effect on their peer was a marginal decrease of less than 0.01)—in other words, a decline, but somewhat less than the “boycotting” schools. (Peer scores have been long declining.)
So, my assumption was somewhat right—it slightly overstated the effect but it rightly identified the likely effect.
Schools that saw a 0.2-point decline: Emory,* Chicago-Kent, UC-Irvine,* UCLA,* Illinois, New Hampshire,* Toledo, Yale*
Schools that saw a 0.2-point increase: Northern Kentucky
*denotes schools that “boycotted” the rankings