Helping the Minoritized Achieve in Academic Science

Re-Evaluation

2015-06-23 12.31.18I have served on a lot of grant panels. In the last year, I served on a grant review panel – a small one for a small foundation that you probably didn’t personally qualify for. I was the only woman of three reviewers on the panel, and each grant had a number of ad hoc reviews provided by experts. For each grant, I submitted made a review, and it took a couple weeks to get them all done. Before I submitted my reviews, I re-evaluated all the proposals to make sure I wasn’t systematically biased against anyone. Because we are all gender biased and racist, I paid particular attention to checking myself on this. And, I ended up changing some of the scores based on the re-evaluations to level them out. Of course there are many reasons with scores could have been different, including if I was tired or hungry when I read the proposal, so I didn’t want those dumb reasons to affect the scores. This final step is one that most people don’t do because of time, but I wanted to do a good job. I did end up raising the scores of more women and of more people with foreign names. I felt like I did a good job, and was proud of my work.

And you know what? I am glad I did the re-evaluations. Here are the reasons:

  1. More women were in the upper half of proposals than would have been if I hadn’t re-evaluated them. Thus, we spoke about more women’s proposals in more detail.
  2. The two very nice gentlemen with whom I was serving were so unconsciously biased against women, it was effing ridiculous. Let me give you an example (the information has been changed to protect the innocent):

Grant 12345, a man:

Scores: Man1: 3, Man2: 3, WOS: 3.

 Comments from the dudes included things like, “Not exactly sure how this will be carried out. Needs more details on the experiments. I cannot tell if this will work.”

Grant 98765, a woman:

Scores: Man1: 3, Man2: 3, WOS: 5.

Comments from the dudes included things like, “Well-described methods, clear proposal, looks like PI will be able to secure federal funding.”

I was shocked. These two proposals had vastly different comments from the two dudes, yet, they gave them the exact same RANKING numbers! On two separate occasions, I convinced one or both of them to change their scores during the discussion. For this example, I actually flat out said, “Please look at the written comments for this person. What exactly is wrong with their proposal? You have nothing negative. So, why is your score the same as the last person who you did have negative comments about?” They could not deny that their rankings were illegitimate, and they changed them.

At another point in the discussion, I pointed out that the ad hoc reviews for a particular woman were biased. For them, this statement went too far. They did not respond well to that. They pointed to a woman ad hoc reviewer to prove to me that the ad hoc reviews were not biased. RED FLAG! If you cannot justify something, pointing to another biased review to justify yourself is not a scientifically good way to prove your point. I told them that it didn’t matter if the reviewer was a woman. Women are just as biased against other women as men. Being a woman does not protect or shield you from sexism. They were not convinced. In the end, I had to write the panel summary, and I had a very hard time. Why? Because they could not point to one thing that they came up with as a panel that was wrong with the proposal. This, to me, stinks of bias. You don’t think anything is wrong, but you just go along with what other people say? That is not the scientific way. All their issues were direct echoes of the ad hoc reviews, which I thought were biased. In fact, they only said positive things in their own personal opinions. Luckily, I did get that person to the funded category, because writing a, “Sorry you didn’t get $$, but we can’t figure out what is wrong,” panel summary is difficult and stupid. At least the letter, which doesn’t say anything is wrong, also doesn’t say, “And you don’t get money for not being wrong.”

There were several other times, I was looking at the comments and thinking, this rating doesn’t jive. More than once, dudes with many negatives were given the same score as women with only positives or far fewer negatives. Dudes just got a leg up. One woman was working with her former advisor still, and that got the old chestnut, “Is this person independent?” but the next dude who was working with their former advisor got, “This is a positive because they will be more likely to be successful working with this bigger group.” These were too far apart in the discussion to successfully combat with logic and reason as I did above, and it wouldn’t have made a difference, but that kind of stuff burns me up.

All in all, the women had to be way better than the men to be ranked equally by these guys, so my re-evaluating of the women only counteracted the lunacy. Also, I was the easy grader for most proposals. I was already seen as the person who gave the highest score (about 1 point higher out of 5 than the guy who gave the lowest scores) on all the proposals, so they didn’t suspect or know that I had re-evaluated and subsequently upgraded a lot of the women.

Based on this experience, I am going to call to all my WomenOfScience friends and male allies: Consider re-evaluating and seeking out your own biases against women and minorities when you review their manuscripts, proposals, or whatever. If you were already going to be nice, act like they walk on water. If you were going to be mean, be 20% nicer. It probably won’t move someone from the unfunded pile to the funded (it didn’t in my case), but it could move someone from the edge into funded, or give someone just a slightly nicer review – you can still be critical without being a total douche.

I will continue to re-evaluate at the end whenever I can, because I think it is the right thing to do. I wish more people would at least be mindful of their biases. To get funded today, you need a champion in the room. Each funded proposal has one. I vow to be the champion in the room for women and under-represented groups. That is what I did. I could have chosen to preferentially fund proposals about beavers (there were no beaver proposals) or a particular school, let’s say Ole’ Miss (there were no Ole’ Miss proposals in the panel), but I chose women. I will continue to choose women. Until there is actual equality.

What do you think? Comment or post here. To get an email every time I post, push the +Follow button.

Comments on: "Re-Evaluation" (3)

  1. What I really wish is that some large organization (NSF or NIH) would take all their reviewer results and analyze them for these biases. Maybe they have already done that, in which case they should put their results about bias in front of reviewers every time they convene a panel. My guess is that the data will strongly support what you observed. I mean, there’s already plenty of studies that back up what you say about bias in various situations, but it would be interesting to have the data on grant refereeing directly from the funding agencies. (And there’s lots of ways to score reviewer comments to see how well they match up with the final ratings, social scientists have done this for years, so this is completely doable.)

    Anyway kudos for your good work on that panel.

  2. I know that some panel discussion revolves around the past success of the group, but sometimes I really wish proposals were at least stripped of names for SOME portion of the review process. I once sat through a discussion of “this proposal isn’t good but the researchers are” where I was the only dissenting voice claiming that the PROPOSAL had to be good as well.

  3. […] I have discussed in previous posts how important it is to speak up and speak out (here, here, here) that I think it is important to point out “bad” behavior. There is sometimes evil […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Tag Cloud

%d bloggers like this: