Comparing Google Scholar's H5 index to Sisk-Leiter citations

After the latest release of Professor Greg Sisk’s scholarly impact measure for law school faculties, Professor Brian Leiter blogged a series of smaller rankings of individual faculty members in different scholarly areas. I thought I’d use the data for a quick look at the difference between measures of scholarly activity. The Sisk-Leiter method is this longstanding project; I thought I’d compare it to Google’s H5 index.

One major barrier to using Google Scholar is that it only works for those who create an account (absent using a time consuming back channel like Publish or Perish). But the two measures do different things.

Google Scholar index covers more works, including far more non-law-related works, than the Sisk-Leiter methodology. Google Scholar includes a number of non-legal and interdisciplinary works. It's a value judgment as to which metric ought to matter--or, perhaps, it's a reason to consider both and acknowledge they measure different things!

Google Scholar gives "credit" for an author being cited multiple times in a single piece; Sisk-Leiter only gives "credit" for one mention. The downside for Sisk-Leiter is that an author who has 12 of her articles published would receive credit in Google Scholar for 12 citations, but only 1 in Sisk-Leiter. On the flip side, an author who cites himself 12 times in a single piece would receive credit in Google Scholar for 12 citations, but only 1 in Sisk-Leiter--and, I think, self-citations are, on the whole, less valuable when measuring "impact."

Google Scholar covers all authors; Sisk-Leiter excludes names omitted in et al. There is a method to help mitigate this concern, but, again, this tends to benefit interdisciplinary scholars in Google Scholar, and tends to benefit (through omission) the more typical sole-author law pieces in Sisk-Leiter. That said, Professor Leiter updated his blog’s rankings with some corrections from Professor Ted Sichelman.

Google Scholar includes references to indexed recognized scholarship; Sisk-Leiter extends to all mentions, including blog posts or opinion pieces typically not indexed in Google Scholar. It's another value judgment as to which metric ought to matter. In this dimension, Sisk-Leiter can be broader than Google Scholar might be.

Sisk-Leiter offers a greater reward for a few highly-cited works; H5 offers a greater reward for breadth and depth of citations. This is a specific measure for H5 in Google Scholar as opposed to Google Scholar more generally. Google Scholar also measures citations in the last five years. But I chose to compare Sisk-Leiter to the Google H5 index instead of the C5 (citations in the last five years) index. H5 measures how many (X) pieces have received at least X citations in the last 5 years. So if you have 10 articles that have each received at least 10 citations since 2013, your H5 index is 10. It doesn’t matter if your 11th piece has 9 citations; it doesn’t matter if one of your 10 pieces has 10,000 citations. It’s a measure of depth and breadth, different in kind than total citations.

In the chart below, I logged the Sisk-Leiter citations and compared them to the Google H5 index. I drew from about 85 scholars who both appeared in a Leiter rankings and had a public Google Scholar profile, and I looked at their Google Scholar profiles this fall (which may mean that figures are slightly off from today’s figures). Google Scholar is also only as good as the profiles are, so if scholars have failed to maintain their profile with recent publications, it may understate their citations. I highlighted in blue circles those identified in the Leiter rankings as age 50 and under.

I included a trendline to show the relationship between the two sets of citations. Those “above” the line are those with higher Sisk-Leiter scores than Google H5 index scores and “benefit", in a sense from the use of this metric over Google H5. Those “below” the line, in contrast, are those who would “benefit” more from the use of Google H5. At a glance, it’s worth considering that perhaps more “pure law” scholars are above the line and more interdisciplinary scholars below it—not a judgment about one or the other, and only a great generalization, but one way of thinking about how we measure scholarly impact, and perhaps reflects a benefit of thinking more broadly about faculty impact. Recall, too, that this chart selectively includes faculty, and that some citation totals vary wildly due to the particular fields scholars write in. The usual caveats about the data apply—there are weaknesses to every citation metric, and this is just a way of comparing a couple of them.

View fullsize

Election counts and recounts

I’ve occasionally tweeted about election margins and recounts. Given a number of (relatively) close elections, refusals to concede, and retracted concessions, I thought I’d offer a little perspective (and a very little math).

All (or, at this point, nearly all) of the change in the margin between candidates in Florida’s Senate, Governor, and Commission of Agriculture races, and Georgia’s Governor race, have arisen because of mere counting of the ballots. There are lots of (here I’ll focus on legitimate, by far the more common) reasons for late-added votes. Provisional ballots could be deemed valid or cured. Vote-by-mail ballots might permissibly arrive after Election Day in some jurisdictions. But these are typically smaller figures. Slow or inefficient counting, or simply more general administrative failures, by election officials might result in added later totals.

To put it obviously, only after the votes have been counted can there be a recount. It’s there that many hold great interest, but it’s there that little changes.

I draw on a few rough figures form past recounts. Minnesota’s Al Franken netted 527 votes in a recount in 2008 in an election with about 2,885,555 votes cast. That turned his 215-vote deficit into a 312-vote victory (post recount and post litigation). The 527-vote change was just 0.018% of all votes cast. (Note that this reflects a percentage of this change compared to all ballots cast. It is not that there were 527 votes that were added for Mr. Franken; votes were added for Mr. Franken and for Norm Coleman, but Mr. Franken gained 527 votes relative to Mr. Coleman.)

Washington’s Christine Gregoire recount netted 390 votes in a 2004 gubernatorial recount in an election with 2,810,058 votes cast. Her opponent Dino Rossi originally held a 261-vote lead, but after two rounds of recounts Ms. Gregoire was declared the winner by 129 votes. The 390-vote change was just 0.014% of all votes cast. (Again, both Mr. Rossi and Ms. Gregoire increased their vote totals in the recount, but Ms. Gregoire increased them at a faster rate than Mr. Rossi, and that 390-vote difference was this percentage.)

Finally, Donald Trump won Wisconsin’s electoral votes over Hillary Clinton by a reported margin of 22,617 votes in 2016. Green Party candidate Jill Stein demanded a recount, which netted Mr. Trump 131 votes and increased his margin of victory to 22,748 among 2,976,150 votes cast. That 131-vote margin reflected 0.0044% of all votes cast. (This was a much more mundane recount in some respects because neither candidate formally challenged the results.)

The long and short of this is, recounts rarely change much in absolute terms. Even in elections with millions of votes cast, the relative change in the margin of victory is extremely low. It was enough to give Mr. Franken and Ms. Gregoire victories, but it was still very little that changed.

One basic reason that’s the case? Recounts recount every ballot—an obvious proposition, of course. But that means a challenging candidate picks up votes as well as his opponent. To successfully change the outcome of an election during a recount, then, you need either extraordinary luck, or you need to establish that there are systematically more votes for your candidate that were “missed” during the first count—mismarked ballots, hanging chads, provisional ballots that were deemed invalid, and so forth.

That can be a high bar. Typically, we’d expect errors like this to be randomly distributed. But as Florida’s 2000 election showed, one county’s procedures (e.g., the use of punch card ballots and a butterfly ballot design) might differ from another’s, which may disproportionately impact one candidate over another to the extent that one candidate’s support resides more heavily in that county. And as some observers of the 2004 Washington race and the 2008 Minnesota race might argue, out-lawyering your opponent can help net a few more votes, too.

Nonetheless, absent evidence like that, or in the event that there is offsetting evidence that may have disadvantaged candidates relatively equally, we would expect little to change in a recount.

Consider Minnesota again. Mr. Franken netted 1 vote for every 5,475 votes ultimate counted, that 0.018% margin. To win, he had to not simply gain votes; he had to gain votes faster than his opponent. And he did so.

And to Washington, Ms. Gregoire fared even worse. She managed to net just 1 vote for every 7,205 ultimately counted.

For both of them, despite these overwhelming odds, the extraordinarily narrow margins of victory—low three-digit margins—helped.

So where do things stand today?

The last update in Florida’s Senate election shows Rick Scott leading Bill Nelson by 12,562 votes among 8,184,631 votes cast. That’s 0.15%, or about 10 times the gains that Mr. Franken and Ms. Gregoire ultimately made—and that would just pull Mr. Nelson into a tie. (Again, however, recall how I opened this post—much depends on whether this is even the final count, as opposed to gains being made in the recount.)

In Florida’s gubernatorial race, among 8,218,682 votes cast, Ron DeSantis leads Andrew Gillum by a margin of 33,684. That’s 0.41%, or would require Mr. Gillum to net 1 vote for roughly every 244 votes cast. (And, of course, this is a rough figure—more ballots would likely be added to the overall total votes cast as “undervotes,” those that failed to register for any candidate, were added to the totals in a recount.)

Florida’s Commissioner of Agriculture, assuredly less watched, may still be instructive once the recount dust settles. Nikki Fried leads Matt Caldwell by 5,326 votes among 8,055,348 votes cast, or a margin of 0.066%.

In Georgia, Brian Kemp needed to secure more than 50% of the vote to avoid a run-off and win outright, and 3,929,937 votes cast, or about 10,875 votes to spare, a margin of 0.28%. (The math looks a little different when looking at a 50% cut-off rather than his position relative to challenger Stacey Abrams, but this works well enough for now.)

In short—the final count matters a great deal for each of these races. Under almost no recount scenario would anyone other than the projected winners win the recount, if recent history is any guide. Only significant election administration errors—failure to count large quantities of votes in select counties, for instance—would be these margins by overcome by challengers, because they’re the kinds of things that were omitted from the count in the first place. That said, there’s a first time for everything, which is probably the news that keeps hope alive for challengers in these races.

"Cyber Interference in Elections and Federal Agency Action"

I have this piece up at the Harvard Law Review Blog, Cyber Interference in Elections and Federal Agency Action. It begins:

Pop quiz: which part of the federal government is tasked with preventing cyber interference in our elections?

Congress has refused to say. We have reached a point of a significant gap between an important federal need and existing federal power. And in the absence of that federal power, federal agencies have stepped into the gap and extended their authority into domains unanticipated by Congress.

Where are they now? Supreme Court clerks, OT 2008

Following up on posts on a ten-year retrospective on the Supreme Court clerks from October Term 2003, October Term 2004, October Term 2005, October Term 2006, and October Term 2007, here's what the clerks from October Term 2008 are doing. This list is probably unreliable and has not been fact-checked in any way, except for the links provided (and these links admittedly often aren't the best source material). As always, please let me know of any errors or corrections.

Chief Justice John G. Roberts

William P. Baude (Yale 2007 / McConnell), law professor at Chicago

Jeffrey M. Harris (Harvard 2006 / Sentelle / Silberman), partner at Consovoy McCarthy Park

Erin E. Murphy (Georgetown 2006 / Sykes), partner at Kirkland & Ellis

Porter N. Wilkinson (Virginia 2007 / Kavanaugh), chief of staff to regents of the Smithsonian

Justice John Paul Stevens

Jessica Bulman-Pozen (Yale 2007 / Garland), law professor at Columbia

Cecilia M. Klingele (Wisconsin 2005 / Black / Crabb (W.D. Wis.)), law professor at Wisconsin

Lindsey Powell (Stanford 2007 / Garland), DOJ

A. Damian Williams (Yale 2007 / Garland), AUSA, S.D.N.Y.

Justice Antonin Scalia

Jameson R. Jones (Stanford 2007 / Sutton), partner at Bartlit Beck

Yaakov M. Roth (Harvard 2007 / Boudin), partner at Jones Day

Moshe Y. Spinowitz (Harvard 2006 / Boudin), partner at Skadden

David C. Thompson (Stanford 2007 / Kozinski), general counsel & chief privacy officer, Grand Rounds

Justice Anthony Kennedy

Ashley C. Keller (Chicago 2007 / Posner), partner at Keller Lenkner

Travis D. Lenkner (Kansas 2005 / Kavanaugh), partner at Keller Lenkner

Steven M. Shepard (Yale 2007 / Kozinski), partner at Susman Godfrey

Christopher J. Walker (Stanford 2006 / Kozinski), professor at Ohio State

Justice David H. Souter

Erin F. Delaney (NYU 2007 / Calabresi), professor at Northwestern

Michael Gerber (Yale 2005 / Leval), AUSA, S.D.N.Y.

Warren D. Postman (Harvard 2007 / W. Fletcher), partner at Keller Lenkner

Noah Purcell (Harvard 2007 / Tatel), Solicitor General of Washington

Justice Clarence Thomas

William S. Consovoy (George Mason 2001 / E. Jones), partner at Consovoy McCarthy Park

Claire J. Evans (Rutgers-Camden 2002 / Sentelle / Chertoff / Simandle (D.N.J.)), partner at Wiley Rein

Jennifer L. Mascott (George Washington 2006 / Kavanaugh), professor at George Mason

Patrick N. Strawbridge (Creighton 2004 / H. Dana (Maine) / M. Arnold), partner at Consovoy McCarthy Park

Justice Ruth Bader Ginsburg

Sue-Yun Ahn (Columbia 2006 / Tatel / Cote (S.D.N.Y.)), senior counsel, Consumer Financial Protection Bureau

Miriam L. Seifter (Harvard 2007 / Garland), professor at Wisconsin

Kevin S. Schwartz (Yale 2006 / Calabresi), partner at Wachtell

Robert M. Yablon (Yale 2006 / W. Fletcher), professor at Wisconsin

Justice Stephen Breyer

Brianne J. Gorod (Yale 2005 / Katzmann / Rakoff (S.D.N.Y.)), chief counsel, Constitutional Accountability Center

Seth Grossman (Yale 2005 / Calabresi / Reinhardt), chief of staff to the President of the University of California

Aileen M. McGrath (Harvard 2007 / Boudin), deputy city attorney, City of San Francisco

Matthew E. Price (Harvard 2006 / Boudin), partner, Jenner & Block

Justice Samuel Alito

Dana Remus (Yale 2002 / Scirica), general counsel, Obama Foundation

Andy Oldham (Harvard 2005 / Sentelle), judge, United States Court of Appeals for the Fifth Circuit

Michael H. Park (Yale 2001 / Alito), partner at Consovoy McCarthy Park

Jack L. White (Pepperdine 2003 / Alito), partner at FH+H

Justice Sandra Day O'Connor

Isaac Lidsky (Harvard 2004 / Ambro, shared with Ginsburg), CEO, ODC Construction

This is by far the most close-knit class of clerks we’ve come across. To wit:

-Three teach at the University of Wisconsin, and two of them, Ginsburg co-clerks, are married to each other. (UPDATE: Two other clerks from OT08, Erin Delaney and Travis Lenkner, are also married to each other.)

-Two Thomas clerks, one Roberts clerk, and one Alito clerk are partners at a boutique founded by at least some of them.

-Two Kennedy and one Souter clerk are also partners at the same boutique founded by at least some of them.

Otherwise, it’s a typical mix of mostly law firm partners, public servants, (including a state SG and a federal appellate judge), and law professors.

MBE scores drop to 34-year low as bar pass rates decline again

On the heels of some good news in recent administrations of the July bar exam comes tough news from the National Conference of Bar Examiners: the Multistate Bar Exam (MBE) scores have dropped to a 34-year low, their lowest point since 1984.

View fullsize

For perspective, California's "cut score" is 144, Virginia 140, Texas 135, New York 133. A bar score of 139.5 is comparable to 2015 (139.9) in recent years. One would have to go back to the 80s to see comparable scores: 1982 (139.7), 1984 (139.2), & 1988 (139.8).

I’d hoped that perhaps qualifications of students have rebounded a bit as schools improved their incoming classes a few years ago; perhaps students are putting more effort into the bar than previous years; or other factors. That appears to not be the case this year.

That said, MBE scores may be slightly less predictive of what will happen with actual bar pass rates. the NCBE has pointed out that the rise of the Uniform Bar Exam has led to a number of test-takers transferring scores to new jurisdictions rather than taking a second jurisdiction’s bar—and, presumably, those who pass in one jurisdiction are much more likely to pass in another jurisdiction (accepting that cut scores can vary in some jurisdictions). The UBE points to a few thousand such transfers last year, at least some of whom may have taken the bar exam. But put against more than 40,000 MBE test-takers, the effect, while real, may be small.

Instead, we’re left to watch as results come in state by state. Tracking first-time pass rates (from jurisdictions that share them so far—ideally, ABA graduates would be a better measure, but this works reasonably well for now), declines have been pretty consistent: New Mexico (-14 points), Indiana (-3), North Carolina (+1), Oklahoma (-8), Missouri (-7), Iowa (-3), Washington (-3), and Florida (-4). But in many of these jurisdiction, pass rates were worse in, say, 2015 or 2016.

We’ll know more in the months to come, but it looks like another year of decline will cause some continued anguish in legal education. The increased quality of law school applicants this year will help the July 2021 bar exam look much better.

Note: I chose a non-zero Y-axis to show relative performance.

A few thoughts on the tempest in Randolph County, Georgia

When election officials in Randolph County, Georgia engaged in the routine practice of election administration, they probably didn’t anticipate a campaign of fearmongering littered with falsehoods to be waged against them.

Randolph County is a rural county with about 4300 registered voters spread over nine precincts. About 55% of registered voters are African-American. Most of the voters—about 60%—are concentrated in just two precincts. Those precincts are 63% African-American, and coincidentally 63% of voters in those two precincts preferred Hillary Clinton in the 2016 presidential election.

There are seven other precincts spread around the county, but they are smaller, and in some cases much smaller. One of those precincts, for instance, has just 73 registered voters.

Keeping polling places open on Election Day can be costly. Voters increasingly vote before Election Day, casting absentee ballots or participating in early voting. Many don’t vote at all. Election officials also noted that these polling places require costly upgrades to make them compliant with disability access laws. They proposed closing these seven of the least-used districts.

Before 2013, election changes like these would have needed prior approval from the Department of Justice under the Voting Rights Act. The Supreme Court in 2013 held that “things have changed” in the South, and that seeking prior approval was no longer needed. Critics of the Supreme Court’s decision have hastily pointed to this episode as demonstration that things haven’t changed and that voter suppression pervasively rears its ugly head in the South. The facts show otherwise.

The American Civil Liberties Union began a public relations campaign demonizing county officials. It led with sinister charges” “7 of 9 precincts” or “75% of polling places” would be closed. “Precincts with 40% of the county’s voters” sounds much less dire, even if that’s more truthful.

The ACLU then claimed that these were racially motivated closings, noting that the county’s population is a majority African-American. But that, too, is a misleading charge. The seven precincts that will close are mostly white and just 42% African-American. Those precincts supported Donald Trump in 2016 with 55% of the vote.

If this is a sinister plot by white officials to suppress African-American voters, it’s the most incompetent plan one could develop. The precincts that will close disproportionately affect white voters and Republicans.

Do the closures affect African-Americans? Certainly. They also affect white & Hispanic voters. But claims of "suppression" suggest that election officials targeted African-Americans, a tough claim to make given that white voters face the brunt of the closures.

Individual precinct closures may disproportionately affect African-Americans. Consider the proposed closure of one precinct with 318 registered voters there, 96% African-American. Then again, we can pick out other precincts, too. There’s one precinct that's 89% white that will close (just 73 registered voters); another that's 83% white will close (just 103 voters).

These facts didn’t stop the ACLU from cherry-picking the 318-voter precinct. They found an uncritical media eager to help spread the misinformation. The Associated Press repeated a series of allegations levied by the ACLU against Randolph County election officials, reading more like a press release than journalism.

This isn’t to say that Randolph County election officials did a very good job. They proposed shuttering these polling places weeks before a statewide general election, and these sites had been used just weeks earlier in the primary election. A longer lead time with better communication to the public might have prevented the fallout.

It might be the case that some voters do have a harder time getting to the polls, or that some voters have to switch to early voting or absentee voting. Those are difficult tradeoffs every election administrator must face, Randolph County included. One hopes they’ve thought through this process more carefully than their action plan suggests.

But administrative mistakes hardly rise to the level of voter suppression, the conscious and deliberate effort to prevent African-American voters from participating in the political process.

That said, the right result was reached--given the lack of information and late notice, it was a welcome decision when the county opted not to close polling locations this fall.

That said, it's worth emphasizing that the political process worked well--no litigation, no order from a judge. Indeed, even in the absence of Section 2 of the Voting Rights Act, a rural county's poll closures became national news. Perhaps for the incorrect hyperbolic reasons described above--but a non-judicial solution nonetheless.

In these anxious political times, fearmongering seems to be the weapon of choice among partisans. But careful attention to detail reveals that the allegations of nefarious plots in Randolph County appear to be no more than less than ideal choices by government officials acting in good faith. Let’s hope that the rhetoric cools ahead of Georgia’s 2018 election.

The demise of the stand-alone law school

Most law schools accredited by the American Bar Association are affiliated with a university. In the last quarter century, we've seen the slow demise of the stand-alone law school. Few are left.

In 1995, Michigan State acquired the Detroit College of Law.

Penn State in 1997 announced a similar plan to create a law school by acquiring the Dickinson School of Law. (Penn State would eventually have law schools at two sites, then split them into two separate law schools under the Pennsylvania State University system.)

In 2010, the Franklin Pierce Law Center affiliated with the University of New Hampshire.

Western Michigan University associated with Thomas M. Cooley Law School in 2014.

William Mitchell College of Law merged with Hamline University School of Law to become Mitchell | Hamline, formally affiliated with Hamline University in 2015.

The recent announcement that the University of Illinois at Chicago would merge with John Marshall Law School is the latest.

(I exclude for-profit schools from this analysis, which come with their own complications. But in 2012, Western State became a part of Argosy University. Savannah Law School recently announced its upcoming closure.)

I also anticipate that someone will point out omissions or errata in my assuredly-incomplete list....

At this point, then, there are only a handful of stand-alone law schools left. The recent news over tenure at Vermont Law School shows that without a university affiliation, weathering tough times can be a significant challenge for stand-alone law schools. How many might remain after the next quarter-century?

Apart from Vermont, there are California Western (in San Diego), Thomas Jefferson School of Law (in San Diego), South Texas College of Law Houston, New England Law | Boston, Brooklyn Law School, New York Law School, and Appalachian School of Law. (Again, please correct any omissions or errata!) (I originally included the University of California Hastings, but given its affiliation with the UC system, perhaps it's simply different in kind and should not be included....)

At this pace, we might expect another couple of closures or mergers in the next few years. And it's simply a demonstration that legal education is changing, and old stand-alone law schools are slowly becoming a thing of the past.

Note: this post has been updated thanks to helpful Twitter feedback and helpful comments!

Amicus Curiae Brief In Support of Neither Party in Baca v. Colorado Department of State

I'm pleased to share a brief I filed in Baca v. Colorado Department of State, a Tenth Circuit case concerning the power of state legislatures to cabin the discretion of presidential electors. You can view the brief here. It's part of a larger project on some archival research on original practices of the states concerning presidential electors--but, alas, the litigation calendar does not await the academic one!

From the Summary of Argument:

The text of the Constitution offers little about the scope of state authority to regulate presidential electors. And there is little judicial precedent about the proper scope of authority of states regulating presidential electors. See, e.g., Ray v. Blair, 343 U.S. 214 (1952). But there are extensive practices in states and in Congress—including practices at the time of the ratification of the Twelfth Amendment—that may help this Court determine the liquidated meaning of these constitutional provisions. Cf. The Federalist No. 37, at 236 (James Madison) (J.E. Cook ed., 1961).

These state and congressional practices reveal three conclusions. First, presidential electors have no right to anonymity when casting their ballots. Second, states have the power to replace presidential electors and levy fines on presidential electors, even after those electors have been selected. Third, Congress holds the power to scrutinize and even reject the electoral votes. In 2017, however, Congress counted Colorado’s electoral votes, and this Court has been asked to revisit a decision reserved to the judgment of Congress. When this Court decides this case, it should interpret the Twelfth Amendment through the practices of the states and of Congress.

"A Future Justice Kavanaugh and Executive Privilege"

Over at Law and Liberty, I have a piece on Judge Brett Kavanaugh and his views on executive privilege. It's titled A Future Justice Kavanaugh and Executive Privilege. From the opening:

If a prosecutor one day decided to indict President Donald Trump, a future Justice Brett Kavanaugh might not be the one to stop it.

Media reports, like this one over at the Washington Post, or this take over at Vox, concerning Kavanaugh’s views on executive power are too quick to assume that he would shield the executive branch during criminal investigations. His writings paint a very different portrait.

Kavanaugh, currently a judge on the District of Columbia Circuit, is a widely discussed candidate to replace Justice Anthony Kennedy on the United States Supreme Court. Kavanaugh has worked extensively in the White House for President George W. Bush. That might lead one to believe that he has a strong view of executive power. But it’s an earlier career experience that may have shaped Kavanaugh’s views, and they are not views that tend to defer to executive power.