Good news from the California bar: the overall bar pass rate rose year-over-year from 43% to 49.6%. Or... did it?

The State Bar of California made a small change to how it calculates the passing rate of bar exam test-takers. In April 2017, it adopted the following change:

It was moved, seconded and duly carried that beginning with the February 2017 administration of the California Bar Examination applicants who did not complete all portions of the examination not be included in the pass/fail statistics published at the time results from the examination are published; and that for an examination to be considered complete, applicant must have achieved a grade of at least 40 on their answers to each question on the examination.

The change is a sensible one: if a test-taker walks out in the middle of the exam, it doesn't seem terribly sensible to include that test-taker as a failure. That's not usually what we'd think about in terms of failure rates; instead, those who sat through the whole exam, answered all the questions, and tried to pass the bar would be the ones whose success rates we'd like to evaluate. A quotation from Karen Goodman on the Committee of Bar Examiners in the Daily Journal was consistent with this: "It seemed like if people did not finish the test, they should not count against the pass rate." (Of course, I suppose, the person did fail!)

At the same time, instituting this new change could make it appear that bar pass rates were higher than they actually were, because the new pass rates are going to be higher than old pass rates due to the change in methodology.

The February 2017 overall pass rate was reported at 34.5%, when under the old methodology it would have been 33.9% (a 0.6-point difference). 78 did not complete the exam

For July 2017, 66 did not complete the exam. That lifted the overall percentage who passed from 49.19% to 49.57%. A California bar representative also informed me that the July 2016 exam had 89 who did not complete the exam, a pass rate of 43.57% v. 43.07%.

(It's worth emphasizing this difference is probably even smaller today because the bar has been shorted from three days to two as of July 2017, making it more likely that more individuals will finish the exam.)

This is a very modest advantage to all schools in reporting their overall pass rates--odds are that one dropout in 200 can bump a school's overall score by a point (when rounded). And it offers a very modest (if slightly deceptive) improvement to the current state of affairs when considering bar passage rates in California. It makes comparisons across years slightly disparate.

But, in an era nearly obsessed with almost any numerical change in bar exam statistics, this one is worth highlighting for future consideration. The true year-over-year comparison is 43.6% to 49.6% (+6 points), or 43.1% [sic; that's the percentage shared with me!] to 49.2% (+6.2 points), not 43.0% to 49.6% (+6.6 points). In future years, the comparison will be easier to make.