Empirical Legal Studies

Replication Problems in Psychology

No doubt partly prompted by a recent spate of academic fraud, Science magazine published results from a study that set out to replicate 100 published psychology studies. An "international team of experts," however, could "reproduce only 36% of original findings." (News coverage here.) While I'm sure many in the psychology field will seek to explain such findings, an inability to replicate 74% of studies published in "top" psychology journals is, at the very least, jarring.

Update: From today's NYT (here).

Modeling Law School Clinics' Influence On Legal Employment Outcomes

Over at The Faculty Lounge blog, Michael Risch (Villanova) dissects a pair of papers that seek to empirically assess whether "law clinic opportunities are likely to improve a school’s graduates’ employment outcomes."

A paper by Jason Yackee (Wisconsin), Does Experiential Learning Improve JD Employment Outcomes?, finds that "there is not much evidence that law schools that provide greater opportunities for skills training have substantially better employment outcomes than do those law schools that provide fewer opportunities.” A subsequent paper by Robert Kuehn (Wash U), Measuring Legal Education's Employment Outcomes, however, concludes that the "expanded empirical analysis finds it is not possible to draw any reliable conclusion from his models about the likely effects of law clinic courses, or other activities like law journal and interschool skills competitions, on employment outcomes, and surely not any negative suggestion about clinic opportunities or participation. The most realistic conclusion from available data is that nationwide models provide inconclusive results, as they do not achieve statistical significance and yield both positive and inverse relationships depending on the year of graduation, control variables, and outliers. In fact, other evidence shows that law clinic experiences are important to potential employers and do aid some students in securing employment."

Setting aside the overheated rhetoric that frequently accompanies discussions about clinical and "experiential" law school offerings, the pair of papers, taken together, illustrates the potential complexities incident to modeling something as complex as legal employment outcomes.

Empirical Assessment of Law Faculty Scholarly Impact

While some may quibble over how best to measure, a recent paper by Sisk (St. Thomas--Minnesota) et al., Scholarly Impact of Law School Faculties in 2015: Updating the Leiter Score Ranking for the Top Third, updates prior work assessing law faculties' scholarly impact. As Brian Leiter observes, "citations to faculty scholarship is, of course, only one metric of scholarly distinction." That said, among Sisk et al.'s interesting findings includes how the scholarly impact rankings for some ("several") schools diverge from US News rankings.

M-Turk's Important Limitations

Dan Kahan (Yale) has been a leading critic of the increasingly-popular "M-Turk" studies. And despite, as Andy Gelman (Columbia--Statistics) notes, "M-Turk’s combination of low cost and low validity [make] it an attractive option for many researchers," such attractiveness, as Kahan points out (here), should be measured due to critical limitations incident to an array of M-Turk's sampling issues. Whether one agrees or disagrees with Kahan, researchers using M-Turk-generated data certainly need to engage Kahan's critique.

Data on Lawyer Malpractice

While legal scholars don't flinch at studying medical malpractice (for one recent example, click here), a similar focus on attorney malpractice has not yet emerged. To this end, in When the Lawyer Screws Up: A Portrait of Legal Malpractice Claims and Their Resolution, Bert Kritzer (Minnesota) and Neil Vidmar (Duke) present initial data that help outline the basic contours of attorney malpractice. While their analyses are largely descriptive, their initial findings point to "essentially two hemispheres of legal malpractice." An excerpted abstract follows.

"In this paper we provide an empirical portrait of legal malpractice claims and their resolution. We draw on a wide range of data sources including reports published by the American Bar Association, reports from individual insurers, and data sets obtained from insurance regulators in Florida and Missouri….

A central finding of our analysis is that there are essentially two hemispheres of legal malpractice paralleling the two hemispheres of the bar identified by Heinz and Laumann in their study of the Chicago bar. In the personal services sector one finds mostly relatively small stakes cases often involving plaintiffs’ personal injury, real estate, family law, and collections and bankruptcy; much of the insurance coverage is provided by mutual insurers started by and/or affiliated with state bars. In the corporate sector stakes tend to be very large, and cases involve corporate matters, corporate litigation, and high stakes areas such as intellectual property and securities; insurance is provided by either by specialized insurers or through brokers who put assemble a group of insurers to cover a large law firm."

More Empirics on Medical Malpractice Reforms

Does Medical Malpractice Reform Increase Physician Supply? Evidence from the Third Reform Wave is the latest installment of a series of papers assessing medical malpractice reforms and their implications for (and on) doctors. In the paper, Myungho Paik (Hanyang Univ.), Bernie Black (Northwestern), and David Hyman (Illinois) find little evidence that caps on non-economic damages influence physician supply. The abstract follows.

"Nine states adopted caps on non-economic damages during the third medical malpractice (“med mal”) reform wave” from 2002-2005, joining twenty-two other states with caps on non-economic or total damages. We study here the effects of these reforms on physician supply. Across a variety of difference-in-differences (DiD), triple differences, and synthetic control methods, we find no evidence that cap adoption predicts an increase in total patient care physicians, in specialties that face high med mal risk (except plastic surgeons), or in rural physicians. Consistent with this analysis, we also find no association between med mal claim rates and physician supply in state and county fixed effects regressions over 1995-2011. We reconcile our results to two prior papers that find evidence of an increase in physicians in high-risk specialties."

Civil Settlements

In A Comprehensive Theory of Civil Settlement, J.J. Prescott (Mich.) and Kathryn Spier (Harvard) dig deeply into the world of settlements and push-back against some conventional wisdom. In particular, they challenge the “'trial versus settlement' trope," and "describe and defend a much richer concept of settlement, amounting in effect to a continuum of possible agreements between litigants along many dimensions."

In addition, the authors "study partial settlements and how they interact with each other in real-world adjudication using new and unique data from New York’s summary jury trial program. Patterns in the data are consistent with parties using partial settlement terms both as substitutes and as complements for other terms, depending on the context, and suggest that entering into a partial settlement can reduce the attractiveness of full settlement."