Nevada has announced it will offer an online bar exam this year. Like other states grappling with bar exam changes, there remain questions I have—security, a new format for test-takers, and fixing the appropriate pass rates. (For what it’s worth, Nevada has consulted with Roger Bolus, who regularly helps state bars think about licensing and cut scores, so that offers promise.)

Karen Sloan reports it as the “first-ever” open book bar exam. Open-book helps some (but not all!) of the security concerns. And the state board of bar examiners praised the open-book format as consistent with what “lawyers do,” looking up applicable law and answering problems.

Really, it’s a new twist on the “Performance Test” component of the bar exam. Such a test gives students a “closed universe” set of facts and law. Examinees answer questions based on these facts. Instead of rote memorization, the test more closely imitates what lawyers do. The new twist? Treat it like an open-book exam.

Most states have a part of their bar exam that looks like the Performance Test—indeed, the Multistate Performance Test is one the National Conference of Bar Examiners offers. But many states have trimmed back its use, and it’s certainly never replaced the bar exam. Why?

A nice piece by Stephen Klein in the Bar Examiner in 1996 summarizes some of these discussion around the Performance Test. Applicants like the test more. Recent lawyers who take the test perform well on it, unlike other components of the bar exam (where rote memorization has faded).

But, the correlation between the Performance Test, other essays, and the multiple choice MBE is high. True, the Performance Test is not extremely highly correlated with those other components. It’s more highly correlated with essays than the MBE. But the correlation between the MBE and essays is also not perfect. In short, all three are related, but they do appear to test different kinds of skills. For all three types of exams, “cross-cutting abilities” like “legal reasoning” appear to drive overall performance.

The Performance Test costs more to administer (like essays), because it requires more labor-intensive grading. It’s tougher to scale without the MBE component as a reference point for more consistent results.

Women tend to perform better on the essay and Performance Test components, so we might expect an essay-only exam to result in a different kind of outcome between men and women than tests that include both a multiple-choice component and an essay component. (There’s no material difference for race or ethnicity.)

So, why haven’t state bars moved toward the Performance Test? Obviously, cost, inertia, and the NCBE’s UBE format are some reasons today, but those can’t entirely explain it.

Instead, I suspect it’s two major reasons. First, state bar licensing authorities remain stuck to the notion that new lawyers need to "know” a number of content areas of the law. The New York State Bar Association, for instance, has pushed this point significantly in what would be a major roll-back to recent bar licensing portability. There’s an expectation that all lawyers simply need to know certain stuff. Memorization is a way to ensure they know it, even if it’s fleeting. And specific content areas being tested force applicants to learn that stuff.

I don’t know that really works the way licensing authorities intend. There’s little evidence that specific substantive law school courses translate into particular bar exam success. Applicants forget much of the memorization shortly after the exam. Many will never practice in most of the areas tested.

Second, if the Performance Test is already highly correlated to the memorization-intensive components of the bar exam, then what value is it really adding? We know that there’s a relationship between low bar exam scores and elevated ultimate attorney discipline rates. Whether that score comes through multiple choice, essays, a two-day exam, a three-day exam, and so on seems marginal. A lower-cost exam that’s more reliable with easier scaling of scores has been attractive to bar licensing authorities. (And, of course, the inertia with the NCBE.)

All this is to say, I’m a fan of the Performance Test on the whole. If we could have two tests, one that costs slightly more but more closely hews to what lawyers do and requires dramatically less memorization, with the present system, the reform seems like the much better bet.

Nevada’s experiment, I hope, will give some reassurance to other states that they, too, could look to alternative bar exam and bar licensing models. We’ll see how it all shakes out in the months ahead.