A literature review of some studies about the bar exam
In recent weeks, there’s been a surge in assertions that the bar exam does “nothing,” is “pointless,” is “worthless,” and so on. Take, for instance, the claims in this op-ed from an Oregon appellate judge, critiquing the claim that the bar acts to protect the public as “completely unfounded,” the bar “does not function to protect the public,” and “I have never heard anyone make a cogent connection between the types of lawyer conduct that harms the public and the screening that occurs via the bar examination.”
These assertions are wrong. I thought I’d start to compile a literature review of studies about the bar exam (one that assuredly can and will be supplemented!).
Here, I seek to compile two sets of studies that may be relevant. (I only include a brief excerpt from each; please do read the entire study for more about the size of the sample, the strength of the inferences, the location where the study took place, and so on!) The first are studies that examine the relationship between bar exam performance and attorney discipline. The second are studies that examine the relationship between bar exam performance and law school grades.
One can challenge these studies, of course (e.g., the strength of the evidence, how significant the effect in any given study is, etc.). And one can still conclude that the bar exam is too costly, too high a barrier to the practice of law, and so on. And one could finally admit these relationships but examine alternative concerns, like the coronavirus pandemic and bar exam logistical problems. But these are, of course, distinct arguments—and I’ve found arguments about the bar exam are too quickly and easily conflated.
For instance, one could accept that there exists a relationship between bar exam performance and attorney discipline—lower bar exam scores correlate with higher ultimate attorney discipline rates, and failing the bar exam at least once correlates with higher ultimate attorney discipline rates. But one could still argue that the risk is too low, or too distantly removed; or, that the increased risk should lead bar licensing authorities to use more powerful tools to prevent that discipline later through better mentoring and oversight. Indeed, in the article I co-authored with Professor Rob Anderson, we identified several of these points—do read the piece!
Additionally, the relationship (at least, a moderate relationship) between law school grades and bar exam performance, I think, tends to undermine the claim that the bar exam is “meaningless”—unless law schools are willing to concede that their own grading is meaningless and employers are willing to concede that reliance on grading is meaningless. (Maybe some will!) While the bar exam is a different test than law school exams (and both are different from the actual practice of law), they do, I think, all tend to test legal analysis, albeit in varied ways. One could still, of course, critique the bar exam as excessively reliant on rote memorization, too costly an investment for law school graduates, and so on. (And while one might ask why the bar exam continues to require graduates of ABA-accredited schools to pass the exam, it’s principally because licensing authorities distrust law schools to maintain adequate admissions, retention, and graduation standards.)
Two more things to consider, too. The first is how to examine the bar exam—one could look at the binary pass-fail outcome, or one could examine a given bar exam score (e.g., distinguishing performance on the bar exam of, say, a score of 145 versus a score of 130). The second is Type I & Type II errors—for instance, the bar admits some people who may end up facing discipline one day, but it does not admit some people who may go their entire careers without facing discipline.
There is much to chew on when examining the relationship between the bar exam and other items, but these, I hope, provide a fruitful starting point for discussions about the bar exam—without the heated (and erroneous) rhetoric that’s too common in these debates.
Please, of course, feel free to contact me if you find a useful study that merits inclusion!
Relationship between bar exam performance and attorney discipline
Leslie C. Levin, Christine Zozula, & Peter Siegelman, A Study of the Relationship Between Bar Admissions Data and Subsequent Lawyer Discipline, LSAC (2013) [archive]
[Ed.: In a Connecticut study, Model 4 drops law school rank and grades as variables, revealing that failing the bar exam is a predictor of future discipline, even when many “character and fitness” variables are controlled.]
Jeffrey S. Kinsler, Is Bar Exam Failure a Harbinger of Professional Discipline?, 91 St. John’s Law Review 883 (2018)
Using bar exam and disciplinary data from Tennessee, this Article substantiates the following theses: (1) The more times it takes a lawyer to pass the bar exam the more likely that lawyer will be disciplined for ethical violations, particularly early in the lawyer’s career; and (2) The more times it takes a lawyer to pass the bar exam the more likely that lawyer will be disciplined for lack of diligence—including non-communication—and/or incompetence.
Robert Anderson IV & Derek T. Muller, The High Cost of Lowering the Bar, 32 Georgetown Journal of Legal Ethics 307 (2019)
Using a large dataset drawn from publicly available California State Bar records, our analysis shows that bar exam score is significantly related to likelihood of State Bar discipline throughout a lawyer’s career. We investigate these claims by collecting data on disciplinary actions and disbarments among California-licensed attorneys. We find support for the assertion that attorneys with lower bar examination performance are more likely to be disciplined and disbarred than those with higher performance.
Kyle Rozema, Does the Bar Exam Protect the Public?, draft 2020
I study the effects of requiring lawyers to pass the bar exam on whether they are later publicly disciplined for misconduct. In the 1980s, four states began to require graduates from all law schools to pass the bar exam by abolishing what is known as a diploma privilege. My research design exploits these events to estimate the effect of the diploma privilege on the share of lawyers who receive public sanctions by state discipline bodies. Lawyers admitted on diploma privilege receive public sanctions at similar rates to lawyers admitted after passing a bar exam for the first decade of their careers, but small differences begin to emerge after a decade, and larger differences emerge after two decades. The estimates suggest that the diploma privilege increased the share of lawyers who received a public sanction within 25 years after bar admission from 4.5 percent to between 4.6 and 6.5 percent.
Relationship between bar exam performance and overall law school grades
Douglass Boshkoff, Phillips Cutright, & Karen Cutright, Course Selection, Student Characteristics and Bar Examination Performance: The Indiana University Law School Experience, 27 Journal of Legal Education 127 (1975)
For example, [the table] shows that 90.9% of students with a cumulative grade point average of 2.8 or higher passed the examination, while only 38.4% of those with an average of 2.0 to 2.3 were successful. The differential indicates a powerful effect of academic performance in law school. Furthermore, this differential is affected little when the other characteristics of graduates with high or low grades are considered . . .
Kristine S. Knaplund & Richard H. Sander, The Art and Science of Academic Support, 45 Journal of Legal Education 157 (1995)
UCLA students with a B+ (83) average in law school are one-tenth as likely to fail the bar exam as students with a C+(73) average.
Linda F. Wightman, LSAC National Longitudinal Bar Passage Study (1998)
Using data from all jurisdictions combined, the logistic regression analyses showed that both adjusted LGPA and LSAT score were statistically significant factors in explaining bar examination outcomes. Another way to evaluate the utility of this model for explaining bar examination outcomes was to examine the correlation between predicted and actual outcomes. For these data, the correlation between predicted and actual pass or fail was .52. (By comparison, the mean correlation between LSAT score and first-year law school average [FYA] was .41 for law schools participating in the 1990-92 LSAC correlation studies. The multiple correlation of LSAT score and UGPA with FYA was .49 for those same schools.)
Linda Jellum & Emmeline Paulette Reeves, Cool Data on a Hot Issue: Empirical Evidence that a Law School Bar Support Program Enhances Bar Performance, 5 Nevada Law Journal 646 (2005)
From the July 1997 examination [of the Virginia bar exam] through the February 2001 examination, the passage rate for students [from Richmond Law] in the bottom half of the class was 51.3%. The fourth-quartile passage rate was 26.0%. In comparison, the passage rate for the top half of the class was 93.9%.
Michael Kane, Andrew Mroch, Douglas Ripkey, & Susan Case, Impact of the Increase in the Passing Score on the New York Bar Examination, National Conference of Bar Examiners (2006)
The high correlations between the two versions of the L-GPA and bar examination scores indicate that there is substantial overlap in what is being evaluated on the bar examination and what is being evaluated in law schools. The strong positive correlation (.63) between the 4-pt L-GPA and bar examination scores indicate that relative performance in law school (independent of the selectivity of the law school) is an important determiner of performance on the bar exam; the 4-pt L-GPA accounts for almost 40% of the variance in bar examination scores. The Index-Based L-GPA has a somewhat higher correlation with bar examination scores (.68) indicating that the strength of the relationship between grades in law school and performance on the bar examination can be enhanced by taking the selectivity of the law school into account; the Index-Based L-GPA accounts for about 47% of the variance in bar examination scores.
The bar examination scores have their highest correlation with the Index-Based L-GPA and their second-highest correlation with the 4-pt L-GPA. So it is clear that performance on the bar examination is strongly related to performance in law school. The correlation of bar examination scores with LSAT scores is fairly high, and the correlation with U-GPA, which has the lowest value of the four correlations, is also reasonably high. Note that U-GPA has a higher correlation with bar examination scores than it has with the LSAT scores. This is somewhat surprising, because the bar examination is taken three or more years after graduation from college, while the LSAT is generally taken closer to the completion of undergraduate education.
Lorenzo A. Trujillo, The Relationship between Law School and the Bar Exam: A Look at Assessment and Student Success, 78 University of Colorado Law Review 69 (2007)
Through research, surveys, and compilation of the resulting data, it became apparent that the single most important predictor of bar passage rate was a student's relative law school class rank.
. . .
This research indicates that neither the LSAT nor undergraduate GPA are as meaningful indicators of success on the bar exam as class rank, which remains the best predictor for success on the bar exam.
Douglas K. Rush & Hisako Matsuo, Does Law School Curriculum Affect Bar Examination Passage? An Empirical Analysis of Factors Related to Bar Examination Passage during the Years 2001 through 2006 at a Midwestern Law School, 57 Journal of Legal Education 224 (2007)
Table 2 demonstrates a strong association between a graduate's final class rank, by quartiles based on final LGPA, and bar examination passage. Graduates of the School of Law who ranked in the first quartile of their law school graduating class passed the bar examination at a 100 percent rate over the five year period of the study. Graduates who ranked in the second quartile of their law school graduating class passed the bar examination at a 95.6 percent rate during the same period. Graduates who ranked in the third quartile of their graduating class passed the bar examination at an 82.6 percent rate and the bar examination passage rate dropped to49.5 percent for those graduates who ranked in the fourth quartile of their graduating class.
The association becomes even more apparent for those graduates who ranked in the bottom 10 percent of their graduating class. Those graduates passed the bar examination at a 27.6 percent rate during the five year period of the study.
Donald H. Zeigler, Joanne Ingham, & David Chang, Curriculum Design and Bar Passage: New York Law School's Experience, 59 Journal of Legal Education 393 (2010)
The bar pass rate of the bottom 10 percent of the graduating class [of New York Law School] was truly abysmal, as is the case at many law schools. The pass rate was sometimes in the single digits and never more than 20 percent.
Derek Alphran, Tanya Washington & Vincent Eagan, Yes We Can, Pass the Bar-University of the District of Columbia, David A. Clarke School of Law Bar Passage Initiatives and Bar Pass Rates - From the Titanic to the Queen Mary!, 14 UDC/DCSL L.Rev. 9 (2011)
As shown in Table 1 there was a difference in bar passage rates on first attempt of 36.8% between students in the upper half of the law school GPA distribution and the bottom half of the GPA distribution. The bottom half of the class was students with a GPA of 2.91 and below. There was a bar passage rate of 92.7% for first and multiple attempts for the top half of the class and 66.4% for the bottom half of the class. This difference in bar passage rates is more pronounced when GPAs are broken out by quartile (Table 2). The bar passage rate on first attempt of the top quartile was86.8% and of the bottom quartile was 25.0%. Over 94.2% of students in the top quartile had passed the bar after their second attempt as compared to 46.0% of students in the bottom quartile.
Nicholas L. Georgakopoulos, Bar Passage: GPA and LSAT, not Bar Reviews, draft 2013
The most striking result of the analysis is the accuracy with which the law school GPA predicts bar passage on the first try. This is visible in the probit models of the first and fourth columns but also in the simple frequency table 5. Graduates with a GPA below 2.6 pass the bar at a less than 10% rate, with two out of 21 students passing. Students with a GPA over 3.2 pass the bar at a well over 95% rate, with three graduates out of 110 failing.
The extraordinary power of GPA to predict bar passage diminishes dramatically for graduates taking the bar for a second time. The second time takers are significantly fewer in number. As a corollary of the high success rate of graduates with GPAs above 3.2 on their first try, very few such graduates appear in this subsample. The success rates, however, do not change nearly as fast as in the subsample of first-time takers. From GPAs of 2.5 to GPAs of 3.1, success rates hover about 50%.
Leslie C. Levin, Christine Zozula, & Peter Siegelman, A Study of the Relationship Between Bar Admissions Data and Subsequent Lawyer Discipline, LSAC (2013) [archive]
Higher law school grades and law school class rank are both negatively associated with discipline risk, but the effect is only statistically significant for grades.
Scott Johns, Empirical Reflections: A Statistical Evaluation of Bar Program Interventions, 54 University of Louisville Law Review 35 (2016)
We found that LSAT and bar exam scores share about 20% of variance . . . 1LGPA and bar exam scores share about 40% of variance . . . and GLGPA and bar exam scores share about 50% of variance . . . . In sum, traditional law school variables share a moderate to strong relationship with bar exam scores but still leave nearly 50% or more of bar exam scores explained by other variables.
Katherine A. Austin, Catherine Martin Christopher, & Darby Dickerson, Will I Pass the Bar Exam?: Predicting Student Success Using LSAT Scores and Law School Performance, 45 Hofstra Law Review 753 (2017)
For first-time bar exam takers, linear regression was conducted to determine whether Texas Tech Law final GPA predicted an individual’s bar exam score. Final law school GPA significantly predicted bar exam performance . . . .
Roger Bolus, Performance Changes on the California Bar Examination: Part 2 (2018)
Based on the results of over 7,500 examinees sitting for the CBX in 2013, 2016 and 2017, the single best indicator for predicting success on the CBX was the final law school GPAs of candidates. This result, while important, is not surprising: students who excel on law school exams would be expected to perform well on the bar as well. Overall, the statistical models developed below which include examinees demographic characteristics, pre-admission credentials and law school performance predicts more than 54 percent of the variability in CBX Total Scale scores. By social science standards, this degree of predictive power is reasonably strong, and well in-line with findings of past efforts in this area.
Amy N. Farley, Christopher M. Swoboda, Joel Chanvisanuruk, Keanen M. McKinley, & Alicia Boards, Law Student Success and Supports: Examining Bar Passage and Factors that Contribute to Student Performance, AccessLex (2018)
More specifically, the post-1L [GPA] model accurately identified 58% of failers, and the most comprehensive post-3L [GPA] model accurately detected nearly 4 out of 5 students who would fail the bar.
Robert R. Kuehn & David R. Moss, A Study of the Relationship Between Law School Coursework and Bar Exam Outcomes, 68 Journal of Legal Education 623 (2019)
Similar to other studies, performance in law school, measured by LGPA, bears the strongest relationship to bar exam outcomes at [Washington University in St. Louis and Wayne State]. Yet LGPA explains only approximately twenty percent of the variability in bar passage rates among graduates. One notable finding at both schools was the correlation coefficient between first-year and final law school grades--above 0.92. This high correlation strongly signals at the end of the first year which group of students is most likely to fail the bar exam and therefore might merit additional assistance over the next two years.