What we don't know about the July 2020 bar exam and Covid-19: a lot

My post last week offering the tentative conclusion that it appears no one contracted the coronavirus during the July 2020 administration of the bar exam languished in obscurity for most of the week until something spurred attention to it, offering voluminous feedback, in some ways the usual assortment of decidedly-helpful to decidedly-unhelpful responses.

There were a couple of tranches of responses I wanted to highlight. The first were a few people encouraged by the results, in particular thinking about how test-takers and bar exam administrators adopted some useful practices ahead of the exam. Test-takers were unusually cautious in self-isolation before the exam to avoid getting turned away from the exam; basic standards like mask or physical distancing requirements go a long way for a relatively short period of time in a room; test-takers traveled by driving in lieu of mass transit to avoid illness. In short, maybe these results shouldn’t be much of a surprise. (They would have been a surprise to me in March, when my robust “might” was all I could muster for an in-person exam!)

The second—and there were much more of these—were those critical of my conclusions. I want to focus on two of the more helpful points to highlight the weaknesses of my post.

The first bucket are questions about whether I asked the right question. As I wrote:

I reached out to several bar licensing authorities to see, as I put it, “whether any person contracted Covid-19 as a result of the administration of the July 2020 bar examination, or if any person contracted it as a result of their traveling to or from or lodging around the time of taking the bar exam.” (Should I have asked a different question? Well, I tried….)

Lots of people suggested I asked the wrong question. For instance, “as a result of” put too much of a causal emphasis, which bar licensing authorities would not necessarily be able to answer—or that made it easier to answer “no.” A positive test shared after the fact could still be reported as one not “as a result of” the exam. Fair critique! My goal was a narrow and specific concern about what we know. But my inquiry could of course have been broader. Additionally, as I noted, “some jurisdictions did emphasize they asked test-takers to disclose if they contracted the coronavirus within 14 days after the exam, and none did so. (That would sweep more broadly than contracting it during the bar exam, of course.)” For a follow-up inquiry (or, if you’re thinking about you, your next inquiry!), I might ask something more broadly, without the causal aspect.

Second, and relatedly about the scope of the question, was whether jurisdictions had contact-tracing protocols in place. Again, some did volunteer the post-exam 14-day reporting period, which, again, suggested to me some contact-tracing protocols in place. But, would it have been more useful to ask—before, during, after the exam—for bar licensing authorities to specify what, if any, contact-tracing protocol they had? Or what their negotiations with the local health officials looked like to do so? Absolutely.

Much of my encouragement came from Colorado's process, because (1) it appears people in the room were all made aware (although I cannot tell how much was word of mouth as opposed to formal notification), and (2) Colorado bar authorities apparently knew the negative results of the test-takers in the room who later took a test as a matter of peace of mind. (That was despite the fact that Colorado's local health authorities said that testing of everyone in the room wouldn't be required because of existing social distancing protocol.)

In short, are there more helpful questions? Absolutely. Is there more evidence we could gather? Definitely. Should we (err, should someone) do so? Totally.

We don’t know the results in a lot of jurisdictions. We don’t know who opted out of the exams. We don’t know how online administrations or administrations split across months look. We don’t know the psychological effect it had on test-takers and how it compares to previous administrations of the exam. We don’t know a lot. Indeed, I only got to seven jurisdictions in my original post! There are a lot of important questions still to ask. (Some I’m working on in the months ahead!)

So… do it! Ask good questions (or, questions better than mine!); engage in good faith with bar licensing authorities and public health officials; determine what we can do going forward!

The most helpful piece, I think, is from Professor Paul Horwitz over at PrawfsBlawg, with this bit I’d like to emphasize:

Voicing ex ante concerns about risks is perfectly understandable. One can hardly wait until after the event to express worries about future risks. And the outcome doesn't mean the concerns about risk were unwarranted. But it seems to me that, at least for those whose arguments were based on academic expertise, or invoked that expertise and appealed to past and ongoing empirical study of the issue, or otherwise invoked a kind of academic or data-driven or scientistic authority in making various arguments, there is an arguable duty to follow up and see what the data ultimately revealed about the accuracy of those warnings.

Claims about the future are often too easy to make. I really like to go back and see how often those (often dire) predictions come true. Sometimes these questions are hard to figure out. This question in particular seems very hard to figure out—hence, a post that opens with “appears,” turns exclusively on the causal questions, and includes a lot of caveats (go back and read all of them!). And, of course, even if true, it doesn’t answer the much bigger questions about the right approach in each jurisdiction (see, e.g., the caveats). It only helps us move marginally forward on one piece of a very complicated puzzle.