Plans to lower California's bar exam score may run up against an impediment: more evidence suggesting that the bar exam score is about where it ought to be.

My colleague Rob Anderson and I recently released a draft study noting that lower bar passage scores are correlated with higher discipline rates, and urging more collection of data before bar scores are lowered.

There are many data points that could, and should, be considered in this effort. The California state bar has been working on some such studies for months. California students are more able than students in other states but fail the bar at higher rates, because California's cut score (144, or in California's scoring 1440, simply 10x) is higher than most other jurisdictions.

Driven by concerns expressed by the deans of California law school, and at the direction of the California Supreme Court, the State Bar began to investigate whether the cut score was appropriate. One such study was a "Standard Setting Study," and its results published last week. It is just one data point, with obvious limitations, but it almost perfectly matches the current cut score in California.

A group of various practitioners looked at a batch of bar exam essays. They graded them. They assessed a score of "not competent," "competent," or "highly competent." They were refined to find that "tipping point," from "not competent" to "competent." (An evaluation of the study notes this is similar to what other states have done in their own standard-setting studies, which have resulted in a variety of changes to bar pass cut scores; and that independent evaluation identified critiques to the study but concluded it was overall sound methodology and valid results.)

The mean recommended passing score from the group was 145.1--more than a full point higher than the actual passing score! The median passing score was 143.9, almost identical to the 144.0 presently used. (The study explains why it believes the median is the better score.)

Using a +/-1 error standard of deviation, the mean score may range from 143.6 to to 148.0; the median score 141.4 to 147.7. All are well short of the 133-136 scores common in many other jurisdictions, including New York's 133. And this study is largely consistent with a study in California 30 years ago when a similar crisis arose over low passing rates, a study I identified in a recent blog post.

So, what to do with this piece of evidence? The researchers offered two recommendations for public comment and consideration: keep the score where it is; or reduce the passing score needed to 141.4 for the July 2017 exam alone. (Note: what a jackpot it would be to the bar test-takers this July if they received a one-time reprieve!) The recommendations nicely note many of the cost-benefit issues policymakers ought to consider--and includes some reasons why California has policy preferences that may weigh in favor of a lower score (at least temporarily). The interim proposal to reduce to a 141.4, one standard error below the recommended median value of 143.9, takes into account these policy considerations. Such a change may be modest, but it could result in a few hundred more bar test-takers passing the bar on the first attempt in California.

Alas, the reaction to a study like this has been predictable. Hastings Dean David Faigman--who called our study "irresponsible," accused the State Bar of "unconscionable conduct" for the July 2016 bar exam--waited until the results dropped to critique the study with another quotable adjective soundbite, labeling the study "basically useless." (A couple of other critiques are non-responsive to the study itself.)

Of course, one piece of data--like the Standard Setting Study--should not dictate the future of the bar exam. Nor should the study of my colleague Rob Anderson and me. Nor should the standard setting study 30 years ago. But they all do point toward some skepticism that the bar exam cut score it dramatically or outlandishly too high. It might be, as the cost-benefit and policy analysis in the recommendations to the state bar suggest, that some errors ought to be tolerated with a slight lowering of the score. Or it might be that the score should remain in place.

Whatever it is, more evidence continues to point toward keeping it roughly in the current place, and more studies in the future may offer new perspectives on the proper cut score.