Helping the Minoritized Achieve in Academic Science

Posts tagged ‘Peer Review’

Criticism – Take it

Julio_Ruelas_-_Criticism_-_Google_Art_ProjectI was chatting a few months ago to a AllyManOfScience who complimented me by saying he uses a lot of the laboratory organizational ideas I present here to organize his lab. (Lab organizational stuff can be found here, here, here, here.) I asked if he had anything to add or modify from what I said, and he added something very interesting. He said that he prefers to hire students who have had some background as an athlete or musician at a high level. He said that people who have done sports or music at a high level are very comfortable with criticism. They have an inherent understanding that even a good performance can still be made better and that critiques are not personal. Critiques are made to make their performance better. I started thinking about it, and I realized that a lot of scientists I know did do sports or music. I was a gymnast who competed at a fairly high level and worked out 24 hours per week to hone my skills. I wasn’t Olympic level, but high enough to be getting a lot of criticism after each routine on a regular basis. HusbandOfScience was a band nerd who taught himself guitar. He spent hours practicing guitar in high school. If you have a good musical ear, you can self-correct, and do not need others to tell you you did it wrong. Other WomenOfScience friends were cheerleaders, synchronized swimmers, and even champion dog show groomers/runners. All of these sports take skills and practice and involve getting criticism.

Science is full of criticism. You have to take it and say thank you. Then ask for more if you want to make it. You do an experiment – you get criticism. You make a figure – you get criticism. You give a talk – you get criticism. You make a poster – you get criticism. You write a paper – you get criticism. You apply for a grant – you get criticism. Over and over and over. It doesn’t stop. It won’t stop. The most famous people in science still get criticism when they submit a paper or a grant – even if they get the paper accepted or grant money a lot easier than you.

If you have a hard time taking criticism, I say practice and get better at it, or leave. You can get better at getting criticism. The first time I got a paper review as a graduate student, I cried. We made the changes and the paper got in. The second time I got a paper review as a graduate student, I cried… OK, so I didn’t learn how to take criticism over night. By the time I was a postdoc, I didn’t cry. I was learning how to take criticism. As a professor, my first couple grant rejections got to me, but after writing 10 proposals and finally getting one funded, I didn’t get so bummed when I didn’t get funded.

Reviews can be too harsh. Sometimes reviews are too harsh, too emotional, or just plain mean. And this sucks. But, your job as a logical scientist is to try to see through the crazy and find the truth in the words. Of course, you are entitled to be pissed off at a mean review or overly harsh or unhelpful critique. But, after you have cooled down, try to figure out what is actually wrong with what you did. Perhaps nothing. Perhaps they misread something that was perfectly clear…but perhaps you could make it clearer. Even bat sh*t crazy reviewer number 3 probably has some point.

There are bad reviews. I don’t want to say that all reviews are equal. I am on the editorial board for a journal, and I serve to find the reviewers and make the editorial decisions. Some reviews are, frankly, emotional. As an editor, I don’t want to see, nor do I care about, your emotions as a reviewer. I also don’t care about your personal opinions about science. I care about facts. Your reviews should be full of science facts. If you think that cats can fly, and that is your scientific opinion, you need to back that up with some references. I am OK with your opinions about the style of the writing as long as you make helpful suggestions to make it a better paper. If your review is emotional and not helpful, I’m not going to take it seriously. You are reviewing a scientific paper – not TROLLING your favorite blog.

So, what do you think? Add your two cents here in a comment, or send me a post. To get an email every time I post, push the +Follow button.

PeerReviewZilla

PeerReviewZillaThere was a recent funny article on “How to be the Perfect Mother” from Huffington Post that was a hilarious look at how society tells us conflicting information about how we should act as mothers. You should go look at it if you are a mother, know a mother, or have a mother. Just go see it.

This article, combined with two recent manuscript reviews coming back, got me thinking about how reviewers also often write conflicting advice for your manuscripts. So, I decided to write a satirical version of a manuscript review as an example.

***Note: any resemblance to reviews you may have received or written are purely coincidental.

Enjoy!

We have read and reviewed the manuscript, “This Science Thing is Important for this Other Thing” by Prof.Science. This manuscript investigates the ScienceThing and its interactions with OtherThing, a very important and understudied topic. This group performed many new experiments that had never been done before and had 6 figures each with A-J panels. Their work was executed well and revealed new information about the interactions of ScienceThing with OtherThing that we never knew before. Their clearly written manuscript had a simulation that modeled the results and showed similar trends suggesting a mechanism.

 

Major Concerns:

In performing these experiments, they used well-tested experimental methods along with specific tests to control for errors. They have used these methods to test for effects of ScienceThing on OtherThing and have quantified the effects. Since these methods are well-tested and accepted in the field, they are not novel. We want only novel experiments even if we cannot interpret the results we get from them. Thus, we suggest that the authors perform all new experiments. Further, did the authors investigate how ScienceThing affected OtherThingII? Only one paper on OtherThingII exists, from the OldFart Group, but it is clearly more important than OtherThing, and it should be explored even though almost no reagents exist for OtherThingII. Unless OtherThingII is also investigated, I do not think this paper is very worthwhile.

The authors display histograms of their work and how ScienceThing affects the OtherThing. It is important to be quantitative and have numerical data. For each histogram, they fit to a Gaussian and report the R-squared value of the fit to the data. They use these fits to discuss the results. Why do they do this? Why not use a simple p-value to the data? Isn’t a student’s t-test done on everything? It is clear that the two distributions do not overlap, so they should report the p-value.

The authors used a toy model to show that the ScienceThing behavior that they see could be due to a minimal number of simple rules. Being quantitative and having models is important. We want more quantitative work and models in this field of science. The simulation has the same trends as the experimental data, but it does not exactly match the data, so the model must be worthless. Why did these authors have a model? They are not theorists or modelers; they are experimentalists. They should remove the model, it detracts from the data.

Without the model, the authors do not have a mechanism. We want all science to be mechanistic. It is not good enough to simply observe something and report what happens. For instance, although their toy model uses 3 simple rules and has the same general trends as the data, they cannot rule out a model with 10 complicated rules. Thus, they have not revealed the mechanism behind the results they see, and thus the impact of the work is lower in my opinion. Until their work becomes more mechanistic, their results are purely qualitative, and the work is not work publishing.

 

Other Issues:

There are a number of sp errs in this manuscript. Don’t they care how they present thmseves? Its like thei didn’t even porrof read before they sent it out. They need to really fix this. There are way too many issues for me to helpfully point out.

They are missing a number of very important citations particularly from the OldFart group, “Science Stuff: A novel Regulator of Nothing,” JSS 1979; “Science Stuff Moves Science Thing,” JSS 1998; and “Science Stuff to Science Thing,” AJSS 2000. These important references about how ScienceStuff is connected to ScienceThing are important and should be added.

Their experimental methods are not good. They didn’t even present them! I suppose they could be in the supplement, but I didn’t read it, so I wouldn’t know. Even if they are in the supplement, they need to have them in the main text. Maybe, once they take out the model, they will have room in this 5-page paper to have detailed methods.

 

In conclusion, after having read this paper, I feel that these results were obvious and could have been guessed from deductive reasoning. Thus, the experiments were not necessary and the results are not novel. Further, to make the results important and novel, the authors would need to perform a number of extra experiments that were not in the original 60 plots presented, and they would need a mechanism, which they have not proven. Overall, it is clear that this study has no value and, thus, I recommend that this paper be rejected.

Anything to add? Post or comment here. Maybe we can add more examples? If you want to get an email every time I post, push the +Follow button.

Sticking Up For Yourself

FEMEN_Calls_for_Sex-BoycottAs I have discussed in prior posts, academic science is full of criticism. Most of the time, the criticism is important and helps you to make your science better. Sometimes, not so much. I still contend that women take much more criticism than their equally-qualified (or less-qualified) male contemporaries, but I haven’t seen any specific studies on it. Have you?

Either way, part of what you have to do in academic science is to stick up for yourself. Whether criticism is constructive or mean, we all need to learn to stick up for ourselves against it. Here are a few places where you have to stick up for yourself and some advice on how to do it. Disclaimer: I am not perfect at this and would love comments or posts with more information from others.

Response to Reviewers of Manuscripts: Reviews on papers are a good place to start with this topic. You will get criticisms in reviews – even National Academy members have criticisms and must respond to them. Here is what I do to respond to reviewers. I find this method both cathartic and productive.

Step 1, I print the reviews. I have a harder time reading things on the screen – especially critiques. As I read the reviews, I write whatever the first response to come to my head is. Sometimes it is easy, like, “Cite this paper,” or, “Emphasize this more.” Sometimes they are more elaborate like, “We can do these experiments: one, two, three, and they will probably take 2 months.” Sometimes my responses are just plain rude, such as, “Did this reviewer even read the manuscript?” or even that old chestnut, “F*ck You!” Yes, I write these all on the paper – dirty words and all. They are my first responses – whatever pops into my head – and they are very useful.

Step 2, I meet with the research team, and we all go over the responses and what we need to do to fix up the manuscript. This is usually major issues, like new experiments. I have others give me their first responses, too, if they want to. We air out everything and figure out how to address all the critiques.

Step 3, I identify the locations in the paper that require correcting and updating and set to do it as soon as possible. This is revising the manuscript. This is the obvious step.

Step 4, I write a response to reviewers. The actual response is very different from the responses in Step 1. For each criticism, I write a point for point response. Sometimes it is easy, like, “We cited this paper to address this concern,” or “We re-wrote this section to emphasize this point and clarify our reasoning.” The hardest responses are ones where you need to rebut the reviewer. None of us is perfect, and sometimes we need to let the reviewer know that their point was actually not correct. No big deal, right? It happens. But, reviewers are in a power position over the authors, and you don’t want to rub them the wrong way. When I have to rebut the reviewer, I make my case very strong with a lot of evidence and references. I have even been known to consult other experts of the field to have conversations about these issues. This is important if the reasoning behind something is unpublished “common knowledge” of the field. Every field has this common knowledge, but most of the time it isn’t published nowhere you can point to easily. If the reviewer is not in the exact same field as you, they might not be aware, so you have to inform them. I don’t just ask them to take my word for it, but I reference real conversations with other experts of the field who I asked if I could name in the response. No other expert has ever said no. I have talked to others, and I think this approach is unusual. No editor or reviewer has ever said it was wrong, so I will keep doing it, if I need to.

Critiques on Grant Proposals: These are more difficult to stick up for yourself because you don’t get to respond to reviewers. But, if you are resubmitting the proposal, you need to go through the reviews and respond to them implicitly within the new proposal. I basically go through the same method of printing the reviews and responding. I also have had co-PIs do the same exercise in a multi-PI proposal, and many found it fun and helpful and emotionally de-stressing. Then, I identify the areas of the proposal that require the changes and re-write based on the reviews.

On my first panel where I served as a reviewer, one of the proposals I was reviewing actually quoted their prior reviews and directly responded in their proposal.  At the agency I was reviewing for, the panel changes every time, so the new reviewers (me) would have no idea that they were responding to critiques except by this method. It was very effective. I haven’t employed this myself, but I do respond to the critiques of prior reviewers.

Even if I am not resubmitting because they do not take resubmissions or the research fit is not right for that program, I still read and try to understand the critiques. Very rarely, I get a really rough review. My first year as an AssistantProfessor, I wrote a grant to a foundation for a young investigator award. One of the reviewers said, “It remains to be seen if [WomanOfScience] can even successfully start a research program.” Ouch! It was really harsh, but it was reality. It was my first year, and it did “remain to be seen,” but it wasn’t for the reviewer to judge in the proposal review. That is what the tenure evaluation is meant to judge. In order to stick up for myself, I called other near peers and had a conversation about it. Swapping stories of jerky reviewers on proposals always makes me feel better before I start the task of trying to make productive lemonade out of their nasty lemons. It also helps me to decide if I should try again at a certain funding agency with the certain idea, or if it is time to move on.

Letter Writers for Promotion: We hope that all our promotions will be wildly successful, and we will all sail over the bar to get tenure, become full, and all other evaluations. But, this really just isn’t the case all the time. These critiques can kill a career, so sticking up for yourself is imperative. Once you go up for tenure, that might be all she wrote, although I know several people who have come back from failing tenure and went back on the market after unsuccessful tenure bids, so it can be done. Most jobs have a “mini-tenure” review process, and this is the time to identify issues and address them. Much like the case against the reviewers of papers, you need to understand the issues, determine how best to address them, and build a case in your favor to make sure you do better at the next promotion evaluation. I suggest doing basically the same process as above.

Step 1 helps you to identify what went wrong and come up with gut instincts as to how to correct them. Step 2 should be done with a team – hopefully you have mentors who can help you. They do not have to be in your department or even at your college. In fact it might be better if they are away from you. It will help you to get an outside prospective from someone who will not also suffer if you do not get tenure and promotion. Your university invested heavily in you, and they want you to succeed. But that also puts pressure on them, and they can lose effectiveness as mentors under this pressure. Going to outside mentors takes bravery. You have to expose your soft under belly to your mentors to allow them to help you. You will have to be vulnerable. So, these are people you will have to trust to have your best interest at heart. Then you will have to go through step 3 – and make the changes and implement the solutions your team have come up with. There isn’t really an mandatory equivalent to step 4 – writing the response – but it may be important to do just that if you don’t get tenure. Having a clear, thoughtful response to critiques is important if you want to challenge the decision or to try for new positions at different institutions.

Other Places: Sometimes we experience critiques in other places such as in email exchanges, in person at faculty or committee meetings, when getting critiques on a proposal or a manuscript from colleagues in person, or even in blog posts 😉 My advice, which I certainly could use a reminder of frequently, is to try not to react immediately. I advise that your initial reactions to bad news, critiques, or even personal attacks should be private. This is often hard to do, but good off-the-cuff reactions take practice to get them right. Public reactions should be carefully planned, if possible. I feel like this is where I fail most. I am obviously much better if I am rested and not hormonal, similar to the uncomfortable conversations post. But, considering that these things can strike without warning, it is hard to always be in the perfect condition to absorb negative comments.

Sometimes the best course is to ignore things. Ignoring something might go against the title of this post which suggests that you should defend yourself, but sometimes it is the best thing. For instance, I recently got an email from a colleague who scolded me because I used a public space for a laboratory end-of-semester party. The public space is adjacent to office space where his students sit. When we went to have the party (all 13 of us), there were only two students (of an office that seats 8) in the office space. Both of these students were  wearing headphones, and it was the second to last day of exam period at 4pm. I didn’t think I was bothering them, and no one said anything at the time when I could have changed anything. Yet, in an email later, I was scolded and told that I needed to ask permission of the students in the office to have my event. I should also mention that I have had these events there previously without scolding, and no one ever said anything before. My first reaction was to write back and defend my actions, but I decided against it. I just ignored the email. I figured if it was really a big deal, he could talk to me in person. I also suspected he threw out the email without much thought. I saw him multiple times the next day without a mention of it from him and with his usual nice self. I think ignoring the situation was the right thing to do there. Sometimes ignoring something is a statement in and of itself. You are saying, “This is not worth my time.”

So, what about you? Do you have any helpful hints on how to respond to criticism. To stick up for yourself in a good way? Any comments or posts would be greatly appreciated! You can receive notice when a new post appears by pushing the +Follow button.

Tag Cloud