An odd and flawed last-minute twist in the California bar exam discussion
My colleague Rob Anderson last night blogged about the strange turn in a recent report from the staff at the California State Bar. Hours ahead of today's meeting of the Board of Trustees, which will make a recommendation to the California Supreme Court about the appropriate "cut score" on the bar exam, new information was shared with the Board, which can be found in the report here. (For background on some of my longer thoughts on this debate, see here.)
The report doubles down on its previous claim that there is "no empirical evidence available that indicates California lawyers are more competent than those in other states." (As our draft study of the relationship between attorney discipline and bar scores in California discusses, we concluded that we lacked the ability to compare discipline rates across states because of significant variances in how state bars may handle attorney misconduct.)
But it's now added a new claim: "Nor is there any data that suggests that a higher cut score reduces attorney misconduct." Our paper is one empirical study that expressly undermines this claim. Rob digs into some of the major problems with this assertion and the "study" that comes from it; his post is worth reading. I'd like to add a couple more.
First, the paper makes an illogical jump: "Nor is there any data that suggests a higher cut score reduces attorney misconduct" to "But based on the available data, it appears unlikely that changing the cut score would have any impact on the incidence of attorney misconduct." These are two different claims. One is an absence of evidence; the other is an affirmative finding relating to the evidence. Additionally, the adjective "unlikely" adds a level of certainty--how low is the probability? And on what is this judgment made? Furthermore, the paragraph is self-refuting: "Given the vast differences in the operation of different states' attorney discipline systems, these discipline numbers should be read with caution." Caution indeed--perhaps not read at all! That is, there's no effort to track differences among the states and control for those differences. (This is a reason we couldn't do that in our study.)
Apart from, as Rob points out, other hasty flaws, like misspelling "Deleware" and concluding that California's discipline rate of 2.6 per thousand is "less than a third" of "Deleware"'s 4.7 per thousand, it's worth considering some other problems in this form of analysis.
At a basic level, in order to compare states based on discipline rates, it must be the case that the other factors do not differ dramatically among states. But if the other factors do not differ dramatically among states, and bar pass score also does not matter, then the states should have roughly equal rates, which they don't.
The figure itself demonstrates a number of significant problems.
First, Figure 7 compares cut score with attorney discipline. But it uses a single year's worth of data, 2015. The sample size is absurdly small--it projects, for instance, the State of Vermont's discipline rate based on a sample of 1 (the total attorneys disciplined in 2015). The ABA has such data for several years, but this report doesn't collect that. In contrast, ours uses over 40 years of California discipline data from over 100,000 attorney records.
Second, the figure doesn't control for years of practice, which can affect discipline rates. That is particularly the case if the cohort of licensed attorneys in the state skews younger or older. We find that attorneys are more likely to face discipline later in their careers, and our study accounts for years of practice.
Third, the figure doesn't recognize variances in the quality of test-takers in each state. In July 2016, for instance, California's mean MBE score was a 142.4, but Tennessee's was a 139.8. Many states don't disclose state-specific MBE data. But two states with similar cut scores may have dramatically different abilities among their test-takers, some with disproportionately higher scores. Our study accounts for differences in individual test-taker scores by examining the typical scores of graduates of particular law schools, and of the differences in typical scores between first-time test-takers and repeaters.
Fourth, the figure treats the "cut score" as static in all jurisdictions, when it has changed fairly significantly in some. This is in stark contrast to the long history of California's cut score. California has tethered its 1440 to earlier standards when it sought applicants to score about 70% correct on a test, so even when it has changed scoring systems (as it did more than 30 years ago), it has tried to hold that score as constant as it can. Other states lack that continuity when adopting the MBE or other NCBE-related testing materials, have changed their cut scores, or have altered their scoring methods. Tennessee, for instance, only five years ago adopted scaling essay scores to the MBE, and failure to do so assuredly resulted in inconsistent administration of standards; further, Tennessee once permitted those with a 125 MBE to pass with sufficient "passing" scores on the unscaled essays. South Carolina at one time required a 125 MBE score, and didn't scale its essays. Evaluating state attorney discipline rates from attorneys admitted to the bar over several decades based on a cut score from the July 2016 test cannot adequately measure the cut score.
Let me emphasize a couple of points. I do wish that we had the ability to compare attorney discipline rates across states. I wish we could dive into state-specific data in jurisdictions where they changed the cut score, and evaluate whether discipline rates changed among the cohorts of attorneys under different standards.
But one of the things our study called for was for the State Bar to use its own internally-available data on the performance of its attorneys on the bar exam, and evaluate that when assessing discipline. The State Bar instead chose this crude and flawed process to demonstrate something else.
Finally, let me emphasize one last point, which I continue to raise in this discussion. Our study demonstrates that lower California bar scores correlate with higher attorney discipline rates, and lowering the bar score will result in more attorneys subject to discipline. But, of course, one can still conclude in a cost-benefit analysis that this trade-off is worth it--that the discipline rates are not sufficient for necessary concern, that they often take years to manifest, that access to justice or other real benefits are worth the trade-off, and so on.
But it is disappointing to ignore or use deeply flawed data about the relationship between attorney discipline and the bar exam cut score in this process, particularly when dumped the night before the Trustees meet to evaluate the issue.