California's leak of bar exam topics should have little if any impact on likelihood of passing

The California State Bar made a big mistake with a lot of questions that remain to be answered. Fortunately, the mistake should have little, if any, impact on anyone’s likelihood of passing the July 2019 bar exam.

From the scant details we know so far, the Bar disclosed to a “number of deans of law schools” the topics that will be tested in the essay portion of the bar exam. That leak took place Thursday, July 25. Late Saturday, July 27, the Bar said it only recen0tly discovered this and emailed all test-takers the same information.

It’s not clear how the leak occurred. It’s also not clear who received the information or what they did with it. Law deans who received this information could have (1) not read the email; (2) read it, but kept its contents confidential; or (3) read it, and shared that information with some test-takers. (3) is the obvious problem. It’s also possible they forwarded the email along to other law school administrators with (1) or (2) in mind, but someone else did (3). (It’s also unclear how the error came to the attention to the Bar or whether any deans revealed this information to the Bar.)

But the obvious concern among test-takers is how this affects their chances of passing the bar. Fortunately, it won’t affect just about anyone’s odds, with a possible exception I’ll get to near the end.

The bar exam is equated and scaled. The essays are scaled to the multistate bar exam (“MBE”), the multiple choice component. I’ve written about what equating and scaling looks like. What it means very roughly is this: there is an opportunity to account for the difficulty or ease of the test itself, and for the higher or lower quality of test-takers, by looking at how those test-takers have done compared to other test-takers on previous administrations of the exam. It’s all anchored to the MBE.

Let me make up a few numbers for the hypotheticals. Suppose there are three bar exam test-takers who get a 150, 140, and 130 on the MBE. Now suppose on the essays they get a 151, 141, and 131. The bar associates the 151 with the 150, the 141 with the 140, and the 131 with the 130, to help scale the scores in recognition that maybe these essays were a bit easier than usual. If the essay scores were 149, 139, and 129, they’d scale them to account for the fact that the essays were a bit tougher than usual.

Suppose that the essays instead are 160, 158, and 156. High compression now—but no difference. They’re still scaled back to the MBE and turn into the equivalent of 150, 140, and 130. These numbers could be 8,000,000,000, 17, and -216: the importance is the relative rank of test-takers against one another, then scaling it back to the MBE.

Revealing the topics of the essays gives everyone two advantages. The first is common (assuming everyone opens the email more than a few hours before the essays Tuesday, July 30.) You know that Question 1 is about Civil Procedure. Everyone knows that and can start from word go addressing Civil Procedure and only Civil Procedure on Question 1. So everyone’s scores may go up a bit.

I say “may,” because what this really does is help the students with the very lowest scores who may have missed the fact that it was a Civil Procedure question in the first place. I think most test-takers know what the call of the question is about. The very lowest scoring test-takers may not. So it may help their scores. But, again, if the scores shift from, say, 151, 141, and 91 to 152, 142, and 97 (disproportionately helping the very lowest scores), it won’t really matter—they’ll still be scaled back to the MBE.

The second advantage is that students can stop studying other, irrelevant topics like Wills & Trusts and focus exclusively on these areas. Again, it should have the same effect: everyone studies the same set of topics, everyone knows them a bit better, everyone scores a bit better. It probably helps the more marginal test-takers over the highest scoring test-takers, but, again, scaling accounts for that if all the boats are rising.

The problem may arise in a narrow set of circumstances. First, suppose the leak did make its way to some (likely very small set of) test-takers, and say it did so very promptly—perhaps as early as 120 hours before the bar exam. Those test-takers immediately stop preparing for irrelevant essays and focus only on relevant essay topics (and the MBE).

Other students only get the information when the Bar emails them, about 58 hours before the essays.

So we have to assume that the three years of legal education, the two months of bar exam preparation, and so on fall to the wayside, and the definitive change in knowledge and understanding occurs in a 62-hour head-start on eliminating some irrelevant subject of study. And in that 62 hours, the students effectively gain additional knowledge to help them for the essays that their peers did not have.

All possible (but, I think, pretty marginal). But it also has to have another effect. It doesn’t really matter if you were a test-taker who was going to get a 157 but for the head start and now gets a 158; it doesn’t really matter if you were a test-taker who was going to get a 92 but for the head start and now gets a 93. In both cases, you’d pass or fail regardless, and it’d have no impact on anyone around you.

Instead, assuming this did confer an advantage (again, possible, but pretty marginal), let’s look at the test-taker on the outside. That test-taker got a 143.9, the closest cut score to passing, but failed. She then looks at the test-taker who got a 144.0 and passed, or maybe a couple who did so (or even those who got a much higher or lower scores)—and these are test-takers who got a 62-hour head-start and eked out a higher score because of that advantage. And they specifically leap-frogged the 143.9—that is, their essay scores in particular would have been lower for the 143.9 but for the 62-hour head start, but they received higher scores, which were then scaled higher. Only for those students whose essay scores improved at a higher rate than others, in a way that affected those at or near the cut score, would experience any material change. (Of course, part of this is context: rather than thinking of a 62-hour head start, it might be useful to say that the person spent only ~1300 hours instead of ~1362 hours thinking about irrelevant topics, assuming two months’ preparation for the bar exam….)

Realize that a lot of things have to happen for this scenario to occur. It’s narrow, sure. But it’s certainly possible. It’s a reason I say “little, if any” impact—this is the scenario where it could change, but it requires a lot to align.

Undoubtedly, test-takers are understandably worried and anxious about any change in information about the bar exam, or any even potential inequities. Law deans will no doubt use this to highlight the incompetence of the Bar and call for an overhaul, perhaps a temporary reduction of the cut score to account for these inadequacies. (What the right solution is, I don’t know. Whether scoring changes come remains to be seen.)

But for the vast majority of law students, the leak should not affect them at all. I hope this gives a small comfort to the thousands of test-takers ahead of Tuesday’s test—you have worked very hard for many long hours, and it is that that will overwhelmingly determine your performance. Hang in there.

I have been lightly updating this post for clarity.

UPDATE, 7-28: Joan Howarth and Rob Anderson have helpfully pointed out that scores might actually increase. By eliminating some subjects, test-takers can now focus on the MBE all the more, too. If that increases MBE scores on Wednesday, then it lifts essay scores that are scaled to those MBE scores, and the overall pass rate could rise. Again, it may be marginal in these last few hours, but it remains possible.

UPDATE, 7-29: I’ve received some thoughtful emails about impact that the disclose may have that I didn’t consider (or didn’t adequately consider) in this post—the mental impact on subgroups, the effect on accommodated test-takers or non-native English speakers, the number of test-takers who have digitally “shut down” and may not learn of this information, and so on. It’s true that all these could have an impact in a way that wouldn’t be reflected in equating and scaling. And these are things very difficult to identify after the fact. Of course, a death in the family or a virus contracted shortly before the exam could have a similar impact—except, of course, (1) this affects all test-takers, and (2) it’s caused by an error from the bar. I appreciate people carefully thinking through the potential effects.