July 2016 bar exam scores improve slightly but remain near all-time lows

The good news for recent law school graduates? The July 2016 pass rates for test-takers will likely increase slightly nationally. As Deborah Merritt recently shared, the mean Multistate Bar Exam score rose from 139.9 in July 2015 to 140.3 in July 2016. Professor Merritt offers a few reasons why scores improved slightly, and I won't add to her good thoughts (despite other possible reasons that may come to light later!).

Given that bar exam scores hit a 27-year low last July, this is surely good news--particularly as the incoming predictors of law students across the nation continued to decline between the entering classes in 2012 and 2013. But there is an important matter of perspective: a 140.3 is still near all-time lows.

View fullsize

The July 2016 score is the second-lowest since 1988 (the lowest being July 2015), and still well off the mark of even the July 2014 score, much less the July 2013 score. In an absolute sense, the score is not good. Indeed, while modest improvements in the bar passage rates in most jurisdictions will be good news for those passing students and for law schools looking for any positive signs, they will not approach the pass rates of three or four years ago.

We should see pass rates from states like North Carolina and Oklahoma soon. As the fall wanes, we'll see more individual jurisdictions and more results from specific schools. And perhaps we'll see if dramatic changes occur in a few places or at a few schools--or whether the change is small and relatively uniform everywhere.

Some mixed results for February 2016 first-time bar exam test-takers

I depicted earlier the 33-year low in February Multistate Bar Exam scores from the February 2016 test. But, as one commenter noted, that doesn't necessarily tell us a whole lot about the first-time pass rate, particularly if there is a glut of repeaters taking the bar inflating the failure rate. Now that results are trickling in, we are seeing declining pass rates across the board among first-time test takers--but, at least in a piece of good news for bar test-takers, the declines are not as dramatically sweeping and across-the-board as we've seen before.

The results are decidedly more mixed, albeit with more in the negative direction than a positive direction. Admittedly, it is important to note that these are only a handful of jurisdictions. Additionally, the February pool of test-takers are much smaller than the July pool of test-takers, which may skew results. The following results are the change in pass rates year-over-year, and I listed them only among first-time test-takers, when the statistics were available. (More states had overall pass rates, but these tend to be more valuable in terms of identifying trends among recent law school graduates.)

Florida, -6 points (February 2015: 64%; February 2016: 58%)

Indiana, -12 points (February 2015: 74%; February 2016: 62%)

New Mexico, -9 points (February 2015: 90%; February 2016: 81%)

Oregon, unchanged (February 2015: 69%; February 2016: 69%)

Pennsylvania, +5 points (February 2015: 69%; February 2016: 74%)

Tennessee, +4 points (February 2015: 64%; February 2016: 68%)

Washington, -4 points (February 2015: 75%; February 2016, 71%)

February 2016 MBE bar exam scores drop to lowest point since 1983

I've written extensively about the bar exam, including the significant decline in bar exam scores, specifically the Multistate Bar Exam, and the corresponding the decline in pass rates in most jurisdictions. The February 2016 results are the fourth consecutive exam to display a significant decline in MBE scores. In fact, it's the lowest score on the February test since 1983--even worse than the July 2015 results, which were the lowest since 1988. Below is a visualization of February test scores since 2005--note the precipitous drop in the last two tests. (I may visualize results since 1983 in the future.)

View fullsize

This will likely mean a decline in pass rates in most jurisdictions, news of which will trickle out over the next several weeks. The decline in scores continues to correlate with declines in student quality, as law schools admitted classes with an increased number of students at risk of failing the bar exam. Whether other factors contribute to the decline remains an open question. But this helps illustrate that the problems are not one-time issues as the result of ExamSoft--they are structural and long-term issues with significant consequences. I'll blog more about this in the near future.

Will states like California lower their bar standards to help schools comply with new ABA mandate?

Unintended consequences are common. One develops a great idea; it takes form; it is discussed and debated; and, finally, it takes effect. But it may result in unintended consequences, it's always been fascinating to think about those unintended consequences. I've extensively discussed unintended consequences of matters such as LSAT administration, accommodated LSAT test-taking, and distortions in law school admissions.

The American Bar Association has moved closer to approving a new accreditation standard. At least 75 of law graduates from an institution must pass the bar exam within two years. It is a much simpler rule than the previous standard, and it holds schools to a higher standard.

Might there be unintended consequences? Many schools right now currently fail that standard. Professor Brian Leiter rightly wonders if schools will focus more on bar prep than other aspects of legal education. It is also likely that many schools will seriously reconsider their class sizes, admissions standards, academic dismissal rates, and transfer students.

But it's also worth noting that not all state bar exams are created equal. Perhaps nothing makes that point so clearly as looking at the passing scores required for the Uniform Bar Exam, a standardized bar exam with a single score, and varying scores required for admission in different states. A 260 will pass in Minnesota or Alabama, while a 280 is required to pass in Alaska or Idaho. My colleague Rob Anderson has identified the varying degrees of difficulty of many states' bar exams. And California is at the top--I've identified how California bar test-takers are more capable than test-takers in other states, but they fail at higher rates because of the difficulty of the bar.

So take a state like California. It is very likely that a number of schools will face serious difficulty meeting this standard--the first time rates for many schools are well below 50%, much less 75%, and even students who retake the test may make it a challenge for the total to pass the 75% threshold.

Some schools may begin to "export" students to jurisdictions with easier exams and higher pass rates--perhaps incentivizing them with stipends on the condition they take the exam in an easier jurisdiction.

But that's a potential unintended consequence that is school-centered. Might there be bar-centered consequences?

Suppose the state bar of California suddenly finds that four or five of its law schools are at risk of losing ABA accreditation. While some may praise that outcome, it's not clear that the state bar would do so. It might be inclined to lower its standards to increase pass rates (more in line with other states) and keep its schools in the ABA's good graces. Other states with particularly difficult bar exams, or with law schools that have significant political clout, may do the same.

Of course, this is speculative. And I make no claim as to whether such decisions would be good or bad--one could think some state bars are too difficult and that the pass rates should be increased, or one could think that the bar should not lower its standards. Instead, it's simply to identify some of the potential consequences that may come about from proposals like this. Only time will tell whether such consequences actually arise.

Heat and light, LSAT scores and bar passage data

If you at all frequently read this blog, you're undoubtedly aware that what largely began as my idiosyncratic thoughts about election law have given way to a significant amount of content on legal education and the bar exam.

Recently, many pixels have been used to discuss the utility of the LSAT, and the relationship between LSAT scores and bar pass rates, which has spurred many larger discussions about the nature of legal education. They are easily discoverable.

Bernie Burk several months ago used the metaphor of heat and light in the midst of some such discussions, which I found quite useful. And I commend to all readers Jerry Organ's comments at the Legal Whiteboard, The Opaqueness of Bar Passage Data and the Need for Greater Transparency. Measured, careful, thoughtful analysis is the analysis I find most useful in such discussions--ones that not only concede limitations, but do not minimize such concessions. I remain deeply grateful for the thoughtful contributors in this space who have spurred me to think carefully and critically on all fronts. And I hope my posts remain useful.

Visualizing the overall bar pass rate declines in 2015 across jurisdictions

In early September, I highlighted the warning signs of bar pass rate declines in several jurisdictions. Shortly thereafter, the NCBE disclosed that MBE scores had hit a 27-year low. Last year, I offered a rough compilation of the decline in overall pass rates, suggesting that ExamSoft was not to blame but that the MBE itself may have contributed to the decline. (Later evidence convinced me that the MBE was likely not responsible for the decline.) By March, we had more granular data for jurisdiction-by-jurisdiction results.

Here are the changes in overall bar pass rates between July 2014 and July 2015. for a handful of jurisdictions that have easily-disclosed top-line data.

View fullsize

The overall declines are far from universal in these jurisdictions, and the median decline is about 2 points. But several jurisdictions did experience overall declines of at least 5 points.

Cobbling together the overall results from July 2013 to July 2015, a two-year change, the trends are fairly stark in most jurisdictions, often joining together significant declines in 2014 with modest declines in 2015.

View fullsize

The entering class profiles for the next few law school classes suggest that these trends will continue, at least to a small degree--the total degree remains something of an open question. How schools react, or how prospective bar examinees react, may further change these projections. And next spring, we'll have the data for the first-time bar-pass rates in these jurisdictions, which will provide a slightly more useful comparison of the overall trends.

California bar exam takers are far more able than others nationwide but fail at much higher rates

The California July 2015 bar results were recently released, reflecting a modest drop in scores, slightly less than other jurisdictions this year. The overall pass rate dropped from 48.6% in July 2014 to 46.6%. The first-time pass rate dropped from 61% to 60%. And among California ABA-accredited schools, the first time rate also dropped a point to 68%.

California is one of the rare jurisdictions that also discloses its statewide mean scaled MBE score. The NCBE discloses the nationwide mean scaled MBE score, which has dropped fairly significantly over the last couple of years. But California has consistently outperformed the nationwide cohort, sometimes rather dramatically.

View fullsize

California's mean scaled MBE score was 142.4 this July, 2.5 points higher than the nationwide average of 139.9. In fact, it's even 0.9 points higher than last year's nationwide average.

The performance of California bar takers is even more impressive given that over 8000 people typically take the July bar among the 50,000 or so nationwide MBE bar exam test-takers. Despite representing over 15% of MBE test-takers, California significantly outperforms the national average.

Pennsylvania (142.2) and Tennessee (139.8) posted lower mean scaled MBE scores. But their pass rates are dramatically higher than California's--71.2% and 64.5%, compared to California's 46.6% in July 2015.

Connecticut and Georgia also disclose the mean scaled MBE scores on a school-by-school basis. I plotted those schools with the California, Pennsylvania, and Tennessee overall results to illustrate how dramatic an outlier California is.

View fullsize

California's high cut score means that many test-takers who would pass the bar in another jurisdiction fail the California bar. Indeed, if the California cut score were closer to Georgia, Pennsylvania, or Tennessee, then the overall pass rate would be around 72%. (It might be even higher in Connecticut.) Indeed, about half of those who failed the California bar in July 2015 would have passed in another jurisdiction.

What the "right" cut score should be is something else. And how the essays bear upon the MBE score. And whether these few states are sufficiently representative. But it's a somewhat useful illustration of some data to note that despite the very low pass rates in California, it's not for lack of relative quality of test-takers.

What happens after a test-taker fails the bar on a first attempt? Some data from Texas

Michael Simkovic and Paul Horwitz have a few thoughts on passing, failing, and retaking the bar exam. I had a few things to add--though, I admit, less specifically about their issues identified!

First, I've blogged extensively about the decline in bar pass rates and the expectation that the declines will continue. A few caveats are in order. Much the top line data I use this cycle is based on overall pass rates; first-time test-taker rates are almost always higher, and first-time test-taker rates at ABA-accredited law schools higher still. But the data we have so far from most jurisdictions is limited to overall pass rates (though, some do disclose more specific information); when the more granular is released this spring, I'll discuss that, too. At the same time, the decline even in overall pass rates is a sign of the decline in overall graduate quality.

First-time pass rates are often the gold standard for a number of reasons. The first, perhaps to most school's chagrin, is the factor in U.S. News & World Report that evaluates a school's first-time pass rate in relation to the jurisdiction's overall rate. But importantly, of course, schools prefer bar pass rate success, and that's most easily identified with first-time pass rates. (Professors Simkovic and Horwitz have some more thoughts about the value or importance of those kinds of things, or about certain schools that are perhaps more vulnerable, which I'll reserve for the time being.) And, loan repayment is expected to begin shortly after graduation, which makes bar passage--and turning one's efforts toward a career is perhaps one of the most important things, particularly if one fails the bar and must take into account opportunity costs of taking time to retake or losing potential time and income in a legal career, or consider the sunk cost of legal education and a prior bar failure, or other such matters.

We have little data, however, about what happens once someone fails the bar exam. There are few longitudinal studies of a specific batch of test-takers. Many jurisdictions simply lump all "repeaters" in a single data set; those who indicate the number of the attempt don't indicate when the previous attempt took place.

But one intriguing study from Texas followed the July 2004 bar exam. The data sets aren't as intuitive as the visuals, so I've offered a couple of ways of interpreting what happened to the folks who took the July 2004 exam--and what happened to them over the next four administrations of the exam.

View fullsize

The overwhelming majority of the 2293 first-time test takers passed on the first attempt. But what happened to the other 474? A majority of that remainder passed on subsequent attempts, but a number dropped out with each subsequent round. 62, for instance, never tried again after the first failed attempt. But 224 passed on the second time around. You can see from the graphic that several more dropped out with each subsequent attempt, until the fourth attempt had just 23 test-takers--and 13 of those passed on their fourth try.

Visualizing the outcomes by each administration offers the following perspective, a kind of narrowing filter:

View fullsize

The study further examined the scores of the test-takers. In each subsequent administration of the test, the scores of the test-takers improved. (On average, of course--some had declines, and some improved far more than the average test-taker.)

View fullsize

Understandably, single-attempt test-takers had by far the highest scores on the first attempt--even though they included 61 test-takers who failed the bar and would never attempt the bar again. And, perhaps predictably, those who only took the bar twice had higher scores on the first attempt than those who would go on to take the bar three or four times. But it's notable that in each group, the results improved. Granted, there's some self-selection in the sense that a few dozen who failed an exam would drop out of the next attempt. And, presumably, those motivated to study with greater discipline are those who are going to take the bar on subsequent occasions. But the suggestion from the scores is that continued time and effort to learn the law will ultimately lead to success--over time.

There are different issues about whether these results are good or bad, or whether retaking the bar exam upon failing is a good or bad idea depending upon circumstances, or whether there are other costs with retaking the bar exam in lieu of other options, and so on--many things that have been discussed elsewhere and will continue to be discussed. But, I find these data points of some interest to track what actually happens among those test-takers from a single administration, and using the data as a starting point for considering what to do with changes in pass rates.

Roundup of recent news about legal education and the bar exam

I've been mentioned in a couple of outlets recently regarding legal education and the bar exam, and thought I'd link to those pieces here and include a brief roundup of news and events in this areas across this blog. You can find me on Twitter @derektmuller.

• Elizabeth Olson, Study Cites Lower Standards in Law School Admissions, New York Times DealBook
• Dina Kleiner, "Are Today's Emerging Lawyers Unfit for the Task?", 88.9 WERS FM

For quick access to categories of interest, check out the Legal education or Bar exam categories on this blog.

Running totals of state bar exams in July 2015 are available here, and an analysis of the 27-year low in the Multistate Bar Exam scores here.

My July 2014 bar exam results wrap-up is here. Summaries of the National Conference of Bar Examiners data in 2014 is here. Finally, my perspective in November 2014 anticipating this and future bar exam score declines: the bleak short-term future for law school bar passage rates.

And I even write about subjects other than the bar exam and legal education! For my scholarly interests--primarily election law--see the remainder of this blog, and my SSRN page.