refereeing

How to Write a Manuscript Review

This one was inspired by a recent conversation in my group meeting.

Generally, the outcome of a review of a manuscript in the physical sciences is one of the following options (I am sure it’s basically the same in the biological and social sciences, and maybe even in the humanities, but I have no direct experience):

a) Accept as is

b) Reconsider with minor revisions

c) Reconsider with major revisions

d) Reject

As a referee, you will be asked to submit a report (to be transmitted to the authors and thus to be written in a collegial manner) along with a recommendation to the editor on the course of action. The recommended course of action is one of the options a) to d). Some journals offer further recommendation subtypes, such as “Accept with optional revisions” vs “Reconsider with mandatory revisions (minor)” vs “Reconsider with mandatory revisions (major)” vs “Reject and recommend transfer to another journal”.  Some have additional options for the referee, such as “I don’t need to see the paper again” or “I need to see the paper again.” But these are all nuances.

Reviews (report plus recommendation) are advisory to the editor. Again, reviews are advisory to the editor. Whether the review is positive or negative, write it so that the editor can understand what has governed your recommendation.

Each referee’s recommendation after a round of review is somewhere between a) and d). Based on these recommendation, the editor makes a  single decision between a) and d) — to accept as is, to invite resubmission with minor or major revisions, or to reject. This single decision is communicated to the authors. The authors will also see the referee reports, but may or may not see the recommendations of individual referees (depending on the field culture and journal), but the authors can usually tell from the content of the reports what each referee recommended. We focus on referee reports and the associated recommendations here.

When do you recommend “Accept as is”?

Option a), “Accept as is,” is usually not recommended by any referee after the first review unless the referee doesn’t give a toss. (It’s okay to “Accept as is” after the first or second revision.) As an associate editor, when I get an adulatory but shallow one-liner after the first review, “This paper is great, publish as is,” I roll my eyes. Such a report is completely useless. It offers me no advice, other than the advice that you as a reviewer didn’t take your job very seriously. Don’t be that reviewer. If you like a paper, your one-liner will not help against a scathing three-page report of another referee who hated the manuscript. If you really like the paper, give the authors something they can use to fight for it.

When do you recommend “Reconsider/accept with minor revisions”?

When you generally like the paper and its conclusion. You think the study is correct, the figures are clear, the conclusions are supported by the data, and the paper is written well. You were able to follow what they did and how, and you have enough information to determine that the technique is appropriate and correctly applied. The minor revisions are usually: missing relevant references (a small number), minor instances of unfortunate wording, some minor tangents that would be interesting to address as they link the paper to the broader field in a way the authors didn’t consider, clarifications in the title or abstract or intro, other clarifications of specific pieces or wording or details in the technique (experimental conditions, theoretical parameters), minor corrections to the figures (e.g., recommendation to choose different colors for better contrast in a 3D plot). Basically, the paper would not be awful to be published “as is” but it could be improved to full awesomeness with edits that are not overly time consuming.

When do you choose “Reconsider with major revisions” vs “Reject”?

Is there plagiarism/duplication of work? If yes, reject, and provide references where the overlapping work has appeared.

Is the paper topically inappropriate for the journal? If yes, then reject, and explain briefly why it doesn’t fit (these are often caught by the editors, so the paper is desk-rejected).

Is the paper not hot enough for the highfalutin journal? If the answer is affirmative, then reject, but please please explain why you think so. A negative one-line review is just as useless as a positive one. The editor can’t do much with your “gut feeling” that the paper is not cool enough for the journal, especially if that’s your only reason to reject the paper. (Unfortunately, what the gut of famous Prof. Greybeard has to say seems to have more weight than the opinions of younger guts). Your gut feeling should in principle be translatable into human speech, such as: all the references are old and there are no new ones, so this work is not timely enough for this journal; most of the references, especially recent ones and/or the ones with similar work, have been published in this other journal instead; the results are straightforward extension of published work and thus of very limited novelty; the results require unrealistic parameters or only occur under a very narrow set of conditions and are thus likely not robust, etc. [see comments for differences among fields].

Are the methods without a doubt inappropriate to address the problem at hand? Then reject. But if the method is one of several and is just not what you would use, that’s not a good enough reason alone to reject the work. Different methods have different strengths and often reveal different facets of the same phenomenon.

Now we come to the tough region.

Is the paper correct? Are the methods appropriate? Is it timely? Is it interesting? Does it present something novel about the world that is not obvious?

If the answer to all these questions is yes, then ask yourself if you can envision this paper being edited so as to become publishable. What would the authors have to do, specifically, to make it suitable for publication?

Does the language need considerable attention? Is the discussion of the techniques/methods unclear? Are the conclusions unclear? Can you write down what specifically is unclear?

If the answer is that you just hate all of the paper, that it’s boring or just awfully written, or that the necessary chances are comprehensive, pervasive in every aspect, then please reject outright. Try to to explain that the paper is far from publishable and that you cannot imagine it becoming publishable within the span of 1-2 revisions; that it would essentially have to become a completely different paper instead because of simultaneous issues with presentation, conclusions, figures, etc. It is much better to reject outright than to 1) torture yourself to try to list all the things that are wrong, 2) make the authors spend a lot of time entering those edits, only to 3) find out that even after all these edits you still think the paper is awful. Rejecting a paper because of pervasive issues is a kindness. For instance, imagine if you were to submit a first draft of a paper written by a second-year graduate student. These drafts usually require extensive edits and the advisor has to make several (many?) layers of corrections in order for it to become suitable to unleash upon the world. Similarly, there is no point in wasting the time of multiple referees “editing by peer review” something that’s as far from publishable as an early draft of a newbie student.

So, when do you say, “Reconsider with major revisions”? When you can envision a finite number of specific things that the paper needs in order to become publishable. Imagine receiving the paper with those revisions perfectly incorporated; if you would then have no problem accepting the paper, then that’s major revisions. Major revisions usually include: significant gaps in cited literature; missing data/figure(s) in order to support a conclusion; missing critical information that prevents a reader from following the exposition or assessing the correctness of the approach; poorly written abstract or conclusion.

How do you write a useful referee report? 

Start with a 2–3-sentence-long paragraph (Hyphen happy! Technically, the first one is a dash.) in which you state, in your own words, what the paper is about, how the authors do what they do, and what the main findings are. This helps show the authors and the editor that you have understood the paper.

Then say clearly, in a single-sentence paragraph,  what your position on the fate of the paper is. Do you feel it’s generally great, but have minor suggestions for improvement or minor but required edits? Do you think it’s inappropriate for publication in the present form, but expect it to become publishable if the authors satisfactorily address the specific problems outlined below? Or do you think the paper is simply not appropriate for publication in This Journal for reasons that are deal-breakers, and concisely explained?

If your are disposed towards rejecting, make sure you state why in a few sentences or a couple of numbered items/paragraphs.

If there are major issues with the paper, give a numbered list of major issues that the authors should address. Be specific about what you want them to do. Remember, if you are a good referee, this should be like a contract: if they do what you ask, you will recommend acceptance. Don’t be that douche who keeps moving the target and asking for new and varied things in subsequent reviews. Follow with a list of minor concerns, like the typos you caught, unfortunate wording, missing units, etc.

If you have identified minor or optional revisions, list them also in a numbered list. If something is optional to consider and you do not require that the authors comply, but just to seriously consider it, then say so.

Happy reviewing!

(See the comments for some differences between fields. Students and postdocs: your own advisor is your best guide regarding established refereeing practices in your field.)

Reviewing Proposals for Foreign Funding Agencies

I spent the first half of this week on travel (fun, exhausting, “the uzhe,” as Eldest would say), followed by a full day of taking kids to various physical exams and dental cleanings, and another day full of meeting my graduate students. Now I have 2.5 weeks before the next trip.

I want to take this opportunity to celebrate a great yet fleeting victory over my work load. As of today, I (temporarily) have no more stuff written by other people (specifically, people who are not my students) in my To-Do folder. In my capacity as an associate editor, I read and then made referrals for the review of a manuscript; I read and then desk-rejected another. I also completed two paper reviews as a referee and submitted a proposal review to a foreign funding agency.

I try to review as much as possible for the journals where I publish often, and I definitely review proposals for the agencies that give me money. The exception is that will generally review stuff where the author is someone whose work I know well, even if it’s in journals or funding agencies I don’t consider.

But, over the last few months, I have been inundated with review requests for proposals to foreign funding agencies: the UK, Swiss, Austrian, and Canadian versions of the NSF. The UK folks alone [Engineering and Physical Sciences Research Council (EPSRC)] have sent me 5 or 6 proposals since February, and they are relentless in sending reminders.

I have to admit I am a little irritated. Reviewing anything takes a lot of time and is generally not a paid activity. It takes away from the time I could be spending on my own work and on my own students. So when expecting people to review stuff, I think there should be at least some theoretical tit-for-tat: I review for journals where I too have my own papers reviewed, I review for agencies where I too have my own proposals reviewed. However, I really find it hard to justify spending my limited time to review proposals for agencies to which I will never be eligible to submit.

Someone might say, “Well, perhaps they don’t have enough experts in their own country to review?” but I would say the EU is probably plenty large to find experts, and I know many solicitations are open to all EU citizens. Furthermore, most US government agencies (in the physical sciences) rarely use reviewers who are not based in the US. So European agencies bugging me in the US to do proposal review for free, and so often, really makes little sense.

As DH says, it’s my fault – I should have never agreed to review for them even once.

What say you, blogosphere? How do you decide when to accept to review papers and proposals? 

Random Bits of Technical Writing

* I am working on a paper that I think has the potential to be a really big deal. It’s so awesome! I am so excited to finish it and submit it that I literally can’t sleep. I sometimes (probably more often than I care to admit) feel like I’m falling in love when it comes to papers or proposals, with butterflies in the stomach from all the anticipation. I can’t get my darling paper out of my head, I keep thinking of the softness of its curves, the color of its data markers, the size of its axis labels… *sigh* …Maybe I need a cold shower.

* There is a colleague whom I met a year or so ago in person, but whose work I have known for a bit longer. His work is technically good, but the papers are not. For some reason, he just can’t write a compelling  narrative or choose the best examples to support the premise. Whenever I read one of his papers, I am thinking — dude, you could have done so much more with this, and there’s always a let-down, a feeling of disappointment when I am done. In the past year or two I have received several of his papers to review, all in lowerish-tier journals; obviously, I am on his preferred-referee list. The first N times (N=3 or 4) I accepted, then tried hard to give detailed constructive advice and feedback. But recently I received his paper N+1, I looked over it and I just couldn’t do it. It’s bad, it’s too little, and the figures look awful. I don’t have time for this, so I declined to review. I would like to help, but I don’t think I am helping through constructive refereeing. He would really benefit from some serious coaching, but he’s not my (or anyone else’s) student or postdoc. I am not sure what to do, probably nothing.

October

I think I might have never been this busy in my life.

Today I was at the Eldest’s swim meet (didn’t need to volunteer, yey!) While all the other parents were sitting around, chatting, playing with their phones and iPads, I had a legal pad, a pen, and a freaking textbook , and was doing undergrad homework problems (i.e. writing up solutions) on my lap, so I can post said solutions before the midterm. Normal people are able to chill over the weekend. Your humble host, not so much. The fact that I was able to find a Starbucks within a 2-min drive greatly  improved the whole experience.

This weekend, I have had three sets of homework solutions to write up (almost done!), and I have to create the midterm exam problems for next week. I also have to finish a brief internal proposal with a colleague and review a proposal for a federal agency. (I still have to go grocery shopping and cook for the week tomorrow. And there are also these kids who seem to expect love, care, and nourishment from me; what’s up with that?)

And this is all so I would have as much prime work time as humanly possible to finish my two proposals by Halloween. Why? Because I just found out yesterday that the  NSF program to which I had been planning to submit one of the proposals, with explicit (written!) encouragement of the program director obtained this summer, has in the meantime changed the program director; the new one is very, very far removed in his interests from the previous guy, so this program is now a no-go. I quickly found a new home for the proposal, but their deadline is 5 days earlier. In all, the proposal which is less done (because its deadline for the original program was later) is, as of yesterday, due sooner than the proposal that’s nearly done.

Of course, when it rains, it pours. There has been so much academic politics drama in the department… So draining. So pointless. So much time wasted by so many people. So much disappointment in colleagues whom I naively used to consider rational and humane, but have since grown to recognize as selfish, vindictive, and manipulative. Egomania, fueled by success in receiving grants and enabled by the money-hungry administration, appears boundless.

Now back to work.

(Ms. Mentor dubbs October the exploding-head-syndrome month.)

Proposal Review Silliness

Lately, I have been reviewing proposals and playing a game with myself  called “Guess how many grants the PI already has based solely on flipping through the proposal to see the formatting.” The correlation is quite pronounced: people who have a reader-friendly layout are universally better funded than those who don’t. When you start reading, you also see that their text flows better (even if the ideas are not necessarily earth-shattering and they end up not funded). Successful grant writers definitely take care of readability, and readability includes thinking about the layout.  Good, reader-friendly layout helps your reviewer think happy thoughts and not dread reading on, as opposed to start hating you with a passion 2 pages in because the gray walls of text gave him/her a claustrophobia attack.

These tips have been mentioned online many times, I am sure, but they never get old: for goodness’ sake, put some space between paragraphs,  go with spacing that’s greater than single (1.1 to 1.2), and break up the text. Ideally, each page will have a figure, a table, or at the very least one or two displayed (as opposed to inline) equations. Look at these three samples — the text from my earlier post, “Musings on Networking” — which format seems the least stifling and the most inviting?

MusPDF1_Page_1

Version 1: single spacing, no break between paragraphs

MusPDF2_Page_1

Version 2: 1.1 spacing, 6 pt break between paragraphs.

MusPDF4_Page_1

Version 3: 1.1 spacing, 6 pt break between paragraphs, plus a figure

All the text is in 11 pt Times New Roman. I am a Times New Roman fan, it’s a classic font and you will never go wrong with it. There are people who like sans serif fonts like Helvetica or Arial, which are just not my cup of tea. But if a funding agency says a font is fine, then it’s fine; have fun with it. Also, I know people will say “But I am constrained to 15 (or however many) pages, I can’t waste space on silly tricks.” Yes, you can and you should. You can purge the fluff and become even more clear and succinct than before, and the reviewer will be happier for it in more ways than one.

— Do give your proposal a title that is short and catchy, but please also be accurate. I got several proposals roughly entitled “Pie in the Sky Is High But I Can Fly” and then one is about the aeronautical engineering of taking off, another is about optimizing sugar-to-flour ratio in pie recipes, and the third is about why the sky is blue and if we can manipulate its color. 

—   There are always fads, hot areas. Once a topic is established as hot, money gets thrown at it, and many people move into the field. However, big groups move faster, and if there are low-hanging fruits, they will be picked by the most nimble. If you are a junior faculty who is just starting with 2 students, you will not be beating a big-named guy who has invented the field, has multimillion-dollar centers funding him to do the work already, and has an army of minions going after the easy pickings. At best, you’ll be just another “me too.” At worst, you will never get any money to do what you are proposing, because groups much bigger and much farther along than you proposed and got funded to do that same exact thing last year and the year before.

When you are just starting up, be mindful of what your strengths are and how fast you can conceivably do something. If you are phenomenally successful at getting money and able to grow much faster than an average new prof, then sure, go ahead, toe to toe with the big guys. But if you are not, then you need to find a niche, something you can do better than others, because of your expertise or how you approach the problems or because no one figured out that some specific aspect may be both important and doable, something that is uniquely yours, not just an obvious question within the latest flavor-of-the-week topic. Be realistic about what you can pull off with the money and personnel you have. And when you identify your niche, then jump on it with all you’ve got.

—  I laughed out loud in my office at a sentence that said something like “In year n of the project, the results will be published in a journal of impact factor at least 10.”

AHAHAHAHAHA! Only 10? Why so low? The stuff is guaranteed to get into Nature! Seriously, people. That’s just amateurish. I can forgive when a graduate student lists 10 papers “in preparation” or “to be submitted to Nature Progeny” on their CV, but a grownup scientist should know better. You can tell me you expect high-impact stuff, but better yet write your proposal in such a compelling fashion that it is crystal clear to me this will be high-impact stuff, in which case I will strongly advocate for funding you. In contrast, the silly silliness of “We’ll totally publish in a Super Duper Glam Mag” is just silly.

Pileup

A colleague once remarked that, past a certain point in one’s career, one could easily spend all of one’s time just reviewing other people’s papers. Truer words have seldom been spoken.

There are several venues where I like to publish and, as a rule, I will accept referral requests from them as long as the manuscripts are within my expertise. I consider it my duty to give back what I expect others to do for my papers. I have long ago stopped reviewing papers from journals I don’t submit to. I know someone has to review for them too, but I simply don’t have the time.

Even so, I always have something waiting in my mailbox. I was down to one referral and was looking forward to having a clear “Papers to Review/Pending” email folder early in the coming week. Then, within three days, I received the following requests that I couldn’t really refuse:

1) A request from a journal where I occasionally publish, for a paper that is well within my expertise and where I know and understand the authors’s work really well.

2) A request from a Reputable Society Journal where I publish very often and from an editor with whom I interact a lot, as he is also the editor on many of my papers by virtue of being the one in charge of that particular topic.

3) A referral from the journal where I am an associate editor. The associate editor making the referral is a senior colleague whom I know and respect and who has been very supportive of my career. The paper is a little more abstract than the average submission, so I can see why he sent it to me and understand he may not have a huge pool of people who could review it.

4) A referral from a journal in which I publish infrequently, but the associate editor is a peer and a friend.

So within 3 days I went from almost having a break from refereeing to now having 5 papers in the queue…  I am hoping there will be at least be some karmic brownie points in all this for me, because otherwise I suspect I may be a sucker.