The Chronicle reports on a new ranking of “Faculty Media Impact” conducted by the Center for a Public Anthropology. The ranking “seeks to quantify how often professors engage with the public through the news media” and was done by trawling Google News to see which faculty were mentioned in the media most often. The numbers were averaged and “and then ranked relative to the federal funds their programs had received” to get the ordering. (Erik Voeten sensibly points out that dividing media citations per faculty member by percent NSF funding amongst participating departments to generate a ranking ”is, to put it mildly, an odd choice”.) As you can see from the screenshot above, the ranking found that MIT came in third overall and that the top unit at MIT was the Sociology Department. This is fantastic news in terms of impact, because MIT doesn’t actually have a Sociology Department. While we’ve known for a while that quantitative rankings can have interesting reactive effects on the entities they rank, we are clearly in new territory here.

Of course, there are many excellent and high-profile sociologists working at MIT in various units, from the Economic Sociology group at the Sloan School to sociologists of technology and law housed elsewhere in the university. So you can see how this might have happened. We might draw a small but significant lesson about what’s involved in cleaning, coding, and aggregating data. But I see no reason to stop there. The clear implication, it seems to me, is that this might well become the purest case of the reactivity of rankings yet observed. If MIT’s Sociology Department has the highest public profile of any unit within the university, then it stands to reason that it must exist. While it may seem locally less tangible than the departments of Brain & Congitive Sciences, Economics, and Anthropology on the actual campus, this is obviously some sort of temporary anomaly given that it comfortably outranks these units in a widely-used report on the public impact of academic departments. The only conclusion, then, is that the Sociology Department does in fact exist and the MIT administration needs to backfill any apparent ontic absence immediately and bring conditions in the merely physically present university into line with the platonic and universal realm of being that numbers and rankings capture. I look forward to giving a talk at MIT’s Sociology Department at the first opportunity.

Update: The page for MIT on the rankings site now contains a Note acknowledging that there’s no Sociology department at MIT. It was added after this post was originally published.

Jim Moody and I are writing an article on data visualization in Sociology. Here’s a picture that won’t be in the final version, but I like it all the same.

Note: The original version of this post, with lots of comments including several follow-ups, clarifications, and further argument from me can be found at OrgTheory. A very useful contribution by Omar Lizardo can also be found there.

Seeing as Fabio has promoted some off-the-cuff remarks I made on Twitter about Critical Realism, I suppose I should say something a little more about it. All the moreso seeing as some anonymous commenters have been getting quite huffy at the very idea that anyone who called themselves an academic could make a dismissive comment without, presumably, devoting themselves full-time to “thoughtful debate and analysis” on the work in question. I have a general and a specific response to that. Speaking generally, online commentary should not be a kind of Markov process where every single contribution must start from scratch with no memory of anything that has gone before. The demand that any particular comment or post provide a full and complete accounting of everything on the topic (before it can count as “thoughtful debate and analysis”) is a hallmark of annoying Internet discussion. My specific response is that some time ago I did in fact devote myself full-time to thoughtful debate and analysis about Critical Realism, for a period of about eighteen months. I read pretty much everything on the topic that had come out until that time, which was a real barrel of monkeys let me tell you. I wrote and published an article on a current debate in the field, and then I moved on to other projects.

My conclusion, then as now, was that Critical Realism is a low-quality, confused, and misleading body of work. It is a justly peripheral branch of 1970s philosophy of science. The philosophical demands it satisfies amongst sociologists could be met elsewhere at much higher quality and far lower cost. In practice it does literally nothing substantive for the work of the sociologists who have taken it up, and I am dismayed to see it gain a foothold in the United States.

Why has CR gotten traction in sociology? In the UK, it was initially attractive mostly to a certain sort of humanist Marxist, especially after it became clear that very strong versions of structuralist Marxism had some unpleasant qualities. The backlash against structure (whether in its Marxist or Parsonian forms) led to various middle-way programs, most notably Giddens’s structuration theory. To many observers this was thin gruel indeed. (They were right about this.) In response, some began to cast around for a way to say social structures were “real” in some strong sense. They sought some philosophically respectable means of defending that idea. Bhaskar’s work, beginning with A Realist Theory of Science, presented itself as an attractive answer. A little later, Margaret Archer retrofitted her already in-progress critique of Giddens with Bhaskar’s vocabulary. This was one of the key pieces of CR’s consolidation in British sociology, as Archer’s theoretical work was a clear and effective attack on the whole structuration program.

In the United States, meanwhile, there seem to have been two channels of diffusion. The first, via Michigan, was through historical sociologists like George Steinmetz and, latterly, Philip Gorski. The second was through religiously-minded sociologists like Chris Smith. I admire the work of these scholars a great deal, by the way. The same goes for Archer, whose work got me reading CR in the first place in 1996. Between them, they jump-started American interest in CR, which up until then had been confined to papers written by a small number of scholars mostly publishing in the Journal for the Theory of Social Behavior.

The diffusion of CR was slightly hampered by the transformation of Bhaskar from fringe philosopher of science to full-blown guru. Having recruited followers in sociology on the basis of his realism, he began to pull the rug out from under them in the late 1990s, first with the merely absurd Plato, Etc and then with the frankly embarrassing From East to West: Odyssey of a Soul (which closed with a final chapter titled “The Dance of Shiva in the Age of Aquarius”). This work, in retrospect, seems like the culmination of the unpleasant cult of personality that grew up around Bhaskar in CR circles in the 1980s and which he seems to have done little to discourage.

CR presents itself in a way that some social scientists—with next to no real background in philosophy—feel gives them just what they need to shore up their empirical research and metaphysical intuitions. You want to be realist in your philosophy of social science? Sure! You want your preferred level of analysis to also be an ontologically emergent level of reality? No problem! You want to talk about social structures as irreducible in some serious-but-not-really-analyzed fashion? You got it. You want your theory to be critical? I mean, who doesn’t, right? Just call yourself a Critical Realist and cite some Bhaskar. After all, he has repeatedly asserted that his work is a “Copernican Revolution” in the philosophy of science—the near-total indifference of everyone else in that field notwithstanding, but never mind that for now.

It’s understandable that sociologists have been susceptible to this sort of thing. The field has long been attracted to strongly expressed emergentist claims. At least since Durkheim, many sociologists have felt that a scientific discipline needs a self-subsistent and “ontologically real” subject matter all of its own. Otherwise, the fear is that the field would become “reducible to” psychology, or worse. Sociological journals are filled with a repeated insistence on the reality of “society” or “the social”, and–conversely—a rejection of “methodological individualism” or “reductionist” approaches to explanation associated with the less social social sciences. No wonder that we’re attracted to a philosophy that seems to “ground” this idea, that guarantees that society or social structure or some supra-individual entities are really real, really. Conveniently, CR also links the allegedly fundamental layered structure of reality with the social organization of academic disciplines round about 1975, with Physics and the bottom, then Chemistry on top of them, then Biology, then Psychology, and then Sociology. (Above that is the Divine bonus layer which you can take or leave as you like.) Each layer has its own emergent properties and causal powers.

The case of emergent properties is a nice example both of what makes CR attractive to some sociologists and of how detached CR became from both the philosophy of science and metaphysics in the 1970s. Many of the claims about emergence characteristic of CR rest on repeated (and probably unwitting) conflation of different senses of what is “emergent” about emergent properties, and what it is to be able to “reduce” something to something else. CR writers typically make the case for their emergence and irreducible properties by way of some stock examples. For instance, they will say that new properties emerge at “higher” levels, as when atoms of hydrogen and oxygen—separately, flammable gases—combine to form water, a molecule that has a property—”wetness”—not “deducible” (CR people will say) from the properties of its component parts. This kind of reasoning is extremely attractive to social scientists for obvious reasons, but it is also very slippery.

Reduction can mean two things. It can refer to the business of translating or defining sentences from one theoretical language in the language of another. For example, we might try to translate statements or theories about water in chemistry, or whatever, into the theoretical language of, say, physics. Or, we might want to “reduce” the propositions and theories of psychology into the language of biology. Whether and to what degree this is possible was the topic of an extensive debate in the ’60s and ’70s (on the “special sciences”, with Putnam, Fodor, and others, involved). The result of this debate was that you generally can’t reduce theories in this way. (Fodor is especially clear on this.)

Second, and separately, there’s the question of ontological reducibility. This is question of whether, independent of your theoretical language, something like a water molecule just is a matter of hydrogen and oxygen (or more fundamental physical entities) in some particular configuration. There’s a further wrinkle here depending whether you are talking about the reducibility of some particular thing or a natural kind.

This distinction between explanatory and ontological emergence is why it doesn’t make sense to talk about about physical properties being deducible from other properties. That can only be true of the linguistic stuff — axioms, theorems, sentences, predicates. CR advocates repeatedly seem to miss this distinction as they want to use examples of alleged ontological irreducibility to shore up claims for the need for a number of distinct disciplines at particular levels of analysis — physical sciences, biology, psychology, sociology, etc. One minute they’re talking about emergence of physical properties in the world (a question of ontology), the next about relationships between theories from this or that discipline at various levels of analysis (a linguistic matter). Running the two together reliably generates an enormous mess.

This is just one of the many ways CR fails to deliver. I am not going to catalog them all here. Again, this isn’t to say that, for example, substantial parts of Archer’s critique of Giddens weren’t correct, or that sociologists shouldn’t—when it comes to developing or testing explanations—rely on “higher level” concepts in their explanations. (Remember: facts about linguistic reduction don’t entail ontological claims about what fundamentally there is.) But there is an awful lot of philosophy that will give sociologists the frankly minimal machinery they need to go ahead and do this work without fear of incoherence. None of the problems CR presents itself as solving are uniquely explicable by (or even originate with) the CR approach itself. Sociologists interested in emergence or macro-level explanations have no need to run together that interest with the specific CR position or view. (This was the point of my old article on Archer, Mouzelis, and CR.) There are large and immediately accessible literatures on all of these topics in philosophy, leading naturally to more technical or specialist work. Consider the SEP article on Emergent Properties, for instance, and the one on Supervenience, or the one on Scientific Explanation, or Scientific Realism. Go from there to, say, Jonathan Schaffer’s Is there a Fundamental Level? (for the metaphysics) or Michael Strevens’ Depth for one take on the philosophy of science, or—for something in parts similar to Bhaskar but far more creative and central—read Nancy Cartwright) on laws of nature. I feel confident in asserting that different sociologists could ally themselves with quite incompatible positions in these debates and much of our work would go on as before.

You’ll notice, if you work through this material, that Bhaskar and Critical Realism are basically not discussed at all. There are good reasons for this, some of which I have mentioned. You might be tempted to feel offended at my dismissal of CR as peripheral. Maybe you think there is some kind of conspiracy in philosophy. “They laughed at Einstein!” you say. (They didn’t really, but never mind.) Well, of course expert scholarly opinion is not infallible. The center of a field doesn’t have a monopoly on insight or valuable argument. But I don’t see anything wrong with asking and taking seriously the views of experts in other fields, especially when CR itself makes such strong claims about its revolutionary effects on those fields. Hence my line on Twitter about Toffler. I bet you’d be at least bemused if you came across a bunch of philosophers building some quasi-sociological research program around the insights of Sociology’s central Tofflerian school. And you might get a little annoyed if they dismissed your efforts to explain that, although sociological insights might be found here and there his work, Toffler is also a somewhat unreliable guide to both the field and the world—his own grand claims to the contrary notwithstanding.

My own view is that sociology needs very little philosophy in order to thrive. Most of the exciting work in sociology has not come from self-conscious theorists seeking to fit the field to some philosophical system, least of all when that system is not even defensible. Much of it, indeed, arises interstitially and out of cross-cutting research groups and programs. This is an imagery of science very much at odds with the carefully-layered picture provided by CR. Indeed, one of the things philosophers of science began noticing in the 1970s was that when two sciences met at a boundary the result typically was not some process of tectonic subduction, with the more “fundamental” science consuming the other by reduction, but rather the appearance of a new science with its own specialist sphere and field-specific concepts and theories—molecular biology, biochemistry, cognitive anthropology, or what have you. CR claims to ground social science with its picture of the fundamental ontology of things, tightly bound to a particular view of what scientific explanation is. And yet in CR-inspired books and articles, it appears more like a lump of cream cheese sitting on top of a bagel: laid on too thick, added after the baking was done, obscuring the stuff you want to chew on, and probably bad for you in the long run. Eat something else.

As the Fall semester is about to begin, here again by popular demand are your invaluable, comprehensive, and wholly accurate twin guides to Interpreting Feedback. First—which, with the exception of a few lines, I didn’t write—is The American Grad Student’s Guide to Interpreting Feedback from Faculty Trained in Britain and Ireland:

Click for a larger version.

And second, its counterpart, The European Grad Student’s Guide to Interpreting Feedback from American Faculty:

Click for a larger version.

June 26th, 2013. I’ve corrected some errors in the dataset. They changes don’t effect the substance of the post. All of these involve merging variant citations to the same work. Notable changes in the graph are the increased prominence of Davidson (1980), van Inwagen (1990), Putnam (1975), and (to a lesser degree), Wittgenstein (1953). I thank Brad Wray for drawing my attention to some of these errors.

Last week’s posts on A Co-Citation Network for Philosophy and Lewis and the Women generated a fair amount of discussion amongst philosophers, and I also got quite a few emails. I want to thank the philosophers who emailed me corrections or spotted duplicates in the item list. I will continue to fix minor errors along these lines. A few people asked whether it would be possible to have a poster-like presentation of the data. Here it is as a PDF file suitable for printing. There is a PNG version as well.

Substantively, I got various requests and suggestions about various ways the data might be further sliced and diced. The two most common responses were, first, for more journals to be added to the list of sources, e.g. leading specialist journals such as Ethics or the British Journal for the Philosophy of Science. And second, people also asked for more detail on what has been happening in the recent past, for instance by focusing on the last ten or even just five years. In response to the first question, and before saying something about the second, it may be worth saying a little bit more about what motivated the original choice of journals and time-frame.

Prestige and Generality

Academic disciplines are complex, dynamic fields where papers, projects, and people compete for attention. There are various research literatures devoted to getting a descriptive sense of what is happening in various fields over time, and—much harder—explaining why things turn out the way they are. Fields are nested and interconnected. Different measures of prominence or quality capture different aspects of the careers of ideas, papers, and people. Prestige orderings for different entities (articles, books, people, journals, presses, departments, subfields, disciplines, universities) are more or less formalized, and are also connected.

In principle all of is supposed to be happening on the basis of quality. At least, when it comes to their own successes, most people want to claim quality is the ultimate explanation. Few really believe it, except perhaps people who have always and only been successful. When it comes to their own failures, reasons other than poor quality immediately suggest themselves. Many of these reasons are quite plausible. From the point of view of any particular career, as Max Weber remarked, “Academic life is a mad hazard”. On the other hand, we also see quite a lot of structure, and competent actors within particular fields typically have a strong practical sense (and often a working theory) of how this structure works and who it favors.

My focus in the Philosophy case was on what’s been happening in high-prestige, generalist journals over the past twenty years. In the long run, what counts as a prestige outlet is up for grabs. In the short run, it’s fixed. Prestigious outlets claim to speak for the field. If they wish to keep doing so, they have to be attentive to where their field is going. Otherwise, the action will move somewhere else, as the field redefines itself and leaves the journal behind. (This is a real possibility in Philosophy, as the volume and specialization of work seems to have been steadily increasing since the 1980s.) I chose four high-impact, high-prestige, nominally generalist journals to work from. The data take all the articles published in these journals between 1993 and 2013 and analyze every item cited in them—about 34,000 citations altogether. The idea is to show what work people in “the field” were talking about (and jointly talking about) over the past twenty years. We pick nominally generalist rather than specialist journals because they’re the most prestigious overall, and they claim to represent the best, most central work in the field.

Work from various subfields in philosophy is not equally represented in these journals. Some is represented in only limited or attenuated ways. Other topics are completely absent. But attempting to define both quality and scope is part of what generalist journals do in academic disciplines. It is presented as the application of judgment, but to those who lose out it can just as easily be read as the exercise of power. It’s important to note that I don’t take a position on this here, I just take it as empirically given. All that matters for our purposes is that these journals are taken to be high-prestige general-interest outlets in the field of English-speaking professional philosophy. I am not judging the quality of subfields or topics. I cannot possibly address here how, for example, pre-career and so-called “pipeline” processes lead women to choose one topic rather than another, or exit one area and enter another.

Rather, I just take it as given that if nominally generalist journals are publishing mostly, say, Philosophy of Mind, Epistemology, and the Philosophy of Language, and those journals are in practice seen as high-prestige generalist outlets—and not abandoned by high-status members of a discipline that has moved on, for example—then what doesn’t get published in these outlets is, in practice, not as central to the field considered from a sociological point of view. The journals were selected to fix a working definition of the high-status end of the field. The 2,200 articles published in them between 1993 and 2013 have passed a stiff test in getting published at all. The citations they contain tell us what was being talked about in these high-prestige outlets. The intrinsic worth of this disciplinary conversation is not something I pretend to judge. But I don’t need to.

Citations are Cheap and Important

Next, why citations? I look at citations because—unlike publishing in these journals, and also unlike getting a job at a top philosophy department—citations are cost-free. An author can in principle cite whomever they take to be a relevant or important interlocutor from the entire history of philosophy, or from any department or journal in the world. The only constraint is the judgment of authors (publishing in these elite places) about whose work is worth talking about. The pattern of these judgments, as revealed in co-citation practices, is what we are interested in. This choice is not really free, of course. In the background is the weight of expectations (from referees, editors, or one’s peers) about who and what to cite. But this is part of how judgments of quality and value are collectively and unequally produced inside academic fields. As I said in the original posts, Philosophy is a particularly interesting case because absolute citation rates are low. Choosing to cite someone carries more weight here than in other fields, and—I argue—makes the co-citation graph a relatively good measure of where the conversational action is in the discipline.

In general, scholarly reputations are made first by being read and talked about informally, catching the attention of influential actors and being judged promising by them. Next comes publications in a relatively high-profile outlet, then by citation in ongoing published conversations. At each stage of this process, we have the opportunity for familiar issues to crop up. For example, perhaps because it is a relatively small and homogeneous field based in large part on excellence in argument rather than quality of data, pure promise—manifested first interpersonally—seems to go further in philosophy than other disciplines. In comparison to most other humanities and social science fields, it is possible (though by no means typical) to get quite far in one’s career on the basis of very little published work. Ultimately, though, published work is necessary, and citations to it grow and consolidate one’s reputation. A general feature of highly cited books or articles is that they gain even more citations as the Matthew Effect takes hold. Other work is ignored, and either moves to a discussion at a more peripheral location or is forgotten.

Are there any Women to Cite?

With the work of Kripke and Lewis dominating the center of the graph, several people asked whether it was all that surprising to see so few women in the dataset, given that there seem to have been so few prominent women working in these dominant areas over the past forty or fifty years. In retrospect, here again it looks like there simply wasn’t anyone to cite. I can’t resolve this question definitively here. My personal prior here is that enough women have worked in these areas for us to have expected at least a few more from the entire history of the field. You may think that all and only the best work has been cited, that all of it is by men, and that’s the end of the matter. But while I am no modal realist, I think the actual configuration of citations in my data is not the only one we might have ended up with.

An alternative hypothesis to pure quality always winning out is that work by women is not cited in high-prestige outlets. Further, this is linked to a cumulative process that reduces the likelihood that talented women will keep working in fields where their work is not discussed. This is not all that controversial an idea in the general literature on citation, although—in part because of negative feedback and endogeneity problems—it is hard to demonstrate cleanly. For the data I have to hand, these issues also get me uncomfortably close to naming and discussing work by people I know personally. This is partly why I was reluctant to narrow the focus to the past five years or the last decade. Narrowing the window for any period makes the data noisier. It becomes impossible to see what’s happening. Coming closer to the present means we focus on people who are working in the field right now. I do not like dealing with a noisy dataset in the context of a touchy subject like whether one’s success is “deserved” in some absolute sense. However, as I’ve repeatedly been asked whether things look different in the past decade, I will make some brief points.

Patterns since 2000

Rather than slicing the data into two ten-year segments and looking at the top 500 cites in each one—something that would make things messy at the lower end of the citation list—let’s take our twenty-year most-cited list and examine articles published in the 2000s alone. This means we are looking at hot papers or books: recently-published items, with relatively little time to accumulate a high citation count, that nevertheless do well enough to make it on to the twenty-year Top 500 list. Such items can of course be authored by very well-established philosophers, or—more interestingly for our purposes—they might be written by younger people making a big early-career impact. Here is the complete list of highly-cited items from the 2000s:

There are 49 articles altogether. Several of the authors are relatively young. Here, “Young” means they got their Ph.Ds in the 1990s or later. Working down the list, these younger people include: Sider, Hawthorne, Stanley (several times), Pryor, DeRose, Fantl, Noe, Hitchcock, Hajek, Cappelen, Merricks, Dowe, Rysiew, O’Connor, Paul, and Heck. (I may have missed one or two: please email me corrections.) There is one woman amongst the lot. The only other item published in the 2000s and written by a woman is by an established philosopher, Lynne Rudder Baker. Some of this recent work by relatively younger men—John Hawthorne (Ph.D ‘91), Ted Sider (Ph.D ‘93), Jason Stanley (Ph.D ‘95), Keith DeRose (Ph.D ‘90) and Jim Pryor (Ph.D ‘97)—even manages to crack the twenty-year Top 100, and in three cases the Top 50. That’s pretty impressive. Notably, the most-cited work by a 90s-cohort Ph.D is not on this list, because it was published in 1996. This is The Conscious Mind by David Chalmers (Ph.D ‘93).

By contrast, we have to go down to joint 435th to find a well-cited paper authored in the 2000s by a—in fact, the only—woman from a comparable Ph.D cohort. This is by Paul (Ph.D ‘99). One of the reasons this is a little awkward is that I am married to this particular data point. I hope I don’t need to emphasize that my wife has had no involvement in the data collection or analysis presented in any of these posts. Incidentally, I didn’t expect there to be so few women in the Top 500. I thought the prevalence of women would be poor, but not terrible.

If we look further down the dataset to the next 500 most-cited items, there are quite a few items written by women from the 1990s-era Ph.D cohort that all of the highly-cited younger men come from. One clear example, brought to my attention by Brian Weatherson, is the case of Delia Graff Fara (Ph.D ‘97). A prominent and very well-respected philosopher, Graff Fara is a full Professor at Princeton working in one of the general areas regularly published in our four journals. She is an obvious case of a very successful woman with a Ph.D from the 1990s who—to Brian’s and many people’s surprise—failed to have an item in the top 500. She is not the only such case. It’s not that her work isn’t cited. She has one paper in the dataset with eight citations, and another with seven. It just isn’t cited enough.

She isn’t alone. Just below the threshold in the data are items by senior women philosophers such as Jennifer Hornsby, Margaret Gilbert, Rae Langton, Karen Neander, Elizabeth Fricker, and Robyn Carston, together with more work by Ruth Millikan, Christine Korsgaard, Linda Zagzebski, Judith Jarvis Thomson, Nancy Cartwright, Susan Haack, and the linguist Angelika Kratzer. Restricting ourselves to items by women with Ph.Ds from the 1990s or later, we find the lower 500 contains work by Nomy Arpaly (Ph.D ‘98), Tamar Gendler (Ph.D ‘96), Miranda Fricker (D.Phil ‘96), Rosanna Keefe (Ph.D ‘97), Helen Beebee (Ph.D ‘96), and Karen Bennett (Ph.D ‘00).

This pattern of results cannot demonstrate, but in my view does suggest, the hypothesis that the process at work here—even in recent years—is not one where women cannot be cited because they simply don’t exist in the relevant subfields favored by journals. Rather, women publish, yet their work is not cited. Citation is not the only measure of success. One can be highly placed without necessarily being highly cited. And, of course, by definition not everyone can be a citation star. But given how easy it is to cite people, this only makes the gender disparity all the more stark. It is very surprising to see only male philosophers from the ’90s cohort so well-represented in the Top 500, and even the Top 100. The 1990s were not the 1950s. And yet essentially none of the women from this cohort are cited in the conversation with anything close to the same frequency, despite working in comparable areas, publishing in comparable venues, and even in many cases having jobs at comparable departments. It is one thing to see such a lopsided pattern in faculty placements, where getting a job is usually a zero-sum competition, and the stakes are very high. To see it in reference lists is worrying, given that compared to the cost of a tenured line, citations are very cheap indeed.