Comment on “Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition”

I just read this worrying paper summarizing a big problem we should all be aware: “Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition“.

I don’t have a perfect solution to change the system for good, but I have an easy patch to help your integrity and the integrity of your group. And I say this because I am very conscious that I am (we all are) weak and when under pressure, the easier person to fool is yourself. This means, that even if you don’t want to cheat consciously, behaviors like p-hacking, ad hoc interpretations and not double checking results that fit your expectations are hard to avoid if you are on your own. So this is the patch: Don’t do things alone. You can fool yourself, but it’s harder to fool your team-mates. And as a corollary, don’t let your students do field work, data cleaning, analysis, etc… alone. Somedays I may be tired and tempted to be more sloppy during fieldwork, for example, but if I have a team-mate with me, it’s easier overcome the situation as a team. In our lab, one way to do this is using git collaboratively. Git tracks all steps of your research since data entry. The first thing we do when we have raw data is to upload it to git and check it ideally at least among 2 of us. This creates a permanent record and avoids the temptation of editing anything there if results are not what you expected later on. Same with data cleaning, and analysis. When those steps are shared and your actions are tracked, it’s easier to be honest. Just to be clear, this mechanism doesn’t work as the threat of a “big brother” that is watching you, is more a feeling of teamwork, where you want to live for the team expectations.

Advertisements

Marie Curie, concessions, and pressure to publish.

I have to admit I didn’t know much about Marie Curie a few days ago (other than the “trivia” facts such as that she discovered the radioactivity and was the first women winner of a Nobel prize). But I just read a book* about her and I really loved it. Oh my god, she was unique in a thousand ways. The book is written by Rosa Montero, and uses Marie Curie’s diary written after Pierre Curie death to talk about very personal things including death, gender balance, society pressures, self-esteem, and many other main topics in life. So it’s not a typical biography, but an excuse to reflect on important things. I won’t go into details, but I highly recommend it.

And while reading the book I found a quote by Pierre Curie that reflects at perfection my actual feeling in science.

“Besides, we must make a living, and this forces us to become a wheel in the machine. The most painful are the concessions we are forced to make to the prejudices of the society in which we live. We must make more or fewer compromises according as we feel ourselves feebler or stronger. If one does not make enough concessions he is crushed; if he makes too many he is ignoble and despises himself”

I do think finding this balance is what kept you (and your science) alive in this world.

Which brings us to the last point. I just discussed a result with my PhD student. It is not significant (p = 0.08), but the effect size is quite big (probability something happening goes from 0.6 to 0.2), but the sample size is small (n < 20). The unavoidable question raised. “It’s 0.08 marginally significant?”, “can we say there is an effect?” My reply was that in a perfect world we would use this data to frame a hypothesis. Then, we would collect 30 more independent data points and test it for real. But the project is almost over, he needs to defend the PhD soon and we are not in a perfect world. So we make concessions. And we will try to publish what we have and cross our fingers hoping that someone else will validate our finding. But we don’t concede too much either, and we should make sure to discuss the result appropriately. A potential large effect size, but very variable and based on a limited sampling size. Or in other words, we will try to avoid the p-value dichotomy once more.

*The book is edited only in Spanish, French, Dutch, and Portuguese… for once, sorry English speakers!

Scrum-like lab meetings

UPDATE: @naupakaz shared a paper explaining a very similar idea via twitter.

Organising successful lab meetings is not an easy task, specially when your group may not have a critical mass of PhD’s and PostDocs. Main aims of the meetings for most labs I’ve been include 1) stay up-to-date on what people is doing, 2) Discuss relevant literature, 3) Give early feedback on peoples projects and 4) discuss techniques, methods or academic culture. And above all, the most important thing for me is that they has the potential to build team spirit.

Despite the above points are all well-aimed, and most times are more or less accomplished, I have also seen some lab meetings fail and people feeling that going is an obligation/waste of time rather than an opportunity. The main reason for that I think is that meetings tend to be no prepared in advance. This is usually associated to overly long meetings and people not being engaged with the topic.

This will be my approach starting next Monday. SCRUM-like stand up meetings every Monday  and Wednesday 9:30 am (sharp!) in my office. Maximum duration 15 minutes (strict!). During the meeting each person will answer this three questions.

  • What I did / learn last week.
  • What I plan to do this week.
  • Which help do I need to accomplish the plan.

In that moment there is no discussion around those questions. If anything needs to be discussed at length, we will schedule a specific meeting for that just with the interested actors. Moreover we will do this in English, not in Spanish, to force new PhD students to get into the habit to talk in english.

This structure has several advantages. Keeps everyone updated and fosters interaction among the different members of the group. Helps you think about what you accomplished /learnt, which should be a positive reinforcement. And finally helps you to plan ahead and be goal oriented. For me as a mentor, allows me to be on the loop on all projects in a fast way, and allows me to act as a facilitator, rather than as an old-school leader. SCRUM people do this everyday, but I think in my lab each people work in a quite different / independent project, so maybe two days a week its fine. However, my IT friends maintain that the way to go is a meeting every morning, so I’ll consider it.

Regarding the other aims of lab meetings, I’ll get advantage of others labs to gain critical mass. EBD runs a monthly Journal club we will attend to discuss papers. This will also make us read broader. We will also join Montse Vilà’s group for getting feedback on projects or discussing methods. This longer meetings will be scheduled when need and will have to requisites. One hour maximum duration and people has to come prepared in advance.

The post is getting lengthy, so I wont go into implementing formal retrospectives every time we publish a paper, but if you are interested in “Agile” development, follow the wikipedia links on SCRUM and Agile.

How to answer to reviewers

This is another of the aspects of doing science that nobody explicitly teach you. The basics are pretty simple to explain (just respond to everything and point by point). You start by mimicking what your mentor does, how other co-authors respond, and how other people respond to your reviewers.  But after seeing different co-authors at work, and specially now that I saw a lot of responses from different people as an editor, there are bad responses, good responses, and those so good, that make your paper fly to the publication stage. Why? The little differences.

1) Be concise (i.e. give a lot of information clearly and in a few words). You can spend some time in formalities and a “thank you” part and a “highlighting the strong points part” is important, but make your case quick and personal. Don’t thank reviewers for “enhancing the paper” because you have too. Thank them for pointing out A or B, which really made a difference. If comments were minor, its not necessary to make a big deal with empty words because you want to be concise. Being personal and not using pre-established “thanks you phrases” helps connecting with the reviewer and sets his/her mood for reading the rest. Also, always briefly highlight the positive things. Editors are busy people, if a reviewer are supportive or partially supportive, bring that up in the response to the editor to put him back in context.

2) Following with conciseness, show that you care about the science. If you did a good work, reviewers do not know your data/analysis as well as you do, so make them trust you by providing details on the decisions you made, and back up all your claims with data and references, not only in the Response to Reviewers, but also in the edited paper. This seems obvious, but I’ve seen several “we don’t agree with this change” without a proper justification.

4) Number your responses. that allow you to refer to previous responses, and avoids repetition. Nobody wants to hear the same justification twice. If your reviewer is not tidy (e.g. do not separate main concerns, from small comments), you should be. Your responses should always flow and for example, you can summary main changes first, and then refer to it when brought up by the reviewer in the middle of other in-line comments that deal with smaller wording issues.

5) Put the focus of the review on the ms, not on the R to R. That means that other than in particular cases you don’t quote the changes in the response, but refer to the lines where the changes are. BUT the real pro-tip is that you highlight the changes in the new ms. Track changes are burdensome and require software specific, but using a different color (I personally like using blue font because red is too contrasting) for the changed sentences in the new ms is a big help for reviewers. This allow both, a smooth read of the full paper, and makes it easier to find the new passages.

Any other tip you use?

Get credit for your reviews: Publons

This posts look like an advertisement, but it is not. I don’t even know who is behind this initiative, but reviewing is an important part of my job, and I think we should get better credit*.

Now I can get some credit easier. Basically, instead of listing in my CV my contributions as a reviewer, I can show them verified at Publons. I can also decide how much information I show about each one. Initially I though that I don’t feel like entering this data manually everytme I do a review, but they allow you to forward the review “thank you” email to them and they add it to your profile, so this makes it quite easy to keep an updated profile. Would be cool that the Journals do that automatically via ORCID, but this is another battle I suspect. And with this tool, you can even calculate pubcreds, if you want!

In any case, the web is easy to use and the guys are very responsive, so I hope this kick off.

*In a nutshell: I advocate for a double blind process, and full disclosure of all names (authors and reviewers) when a decision is made. Until this happens, I sign my reviews.

Short guideline on multi-authored papers.

After being on the two sides of the story (first author and one more of dozens of co-authors) I already made a few errors someone may find useful to know, specially since multi-author papers (more than 10-15 authors from different institutes) are becoming normal (and I am not judging if this is good or bad*, is just happening).

1. Talk about co-authorship early on, but with conditions. This things should be talked at the beginning of the collaboration, because there is nothing more awkward than someone thinking that he/she is coauthoring a paper, while the lead author thinks that he/she is a data provider. However, do not grant co-authorship before even starting the project. Make clear that someone will be a co-author if his/her contribution is [fill in here your expectations] (e.g. the data provided ends up being critical for the paper, you are engaged on ms writing, etc…). By clear I meant very clear.

2. Establish feed-back points. This one is very tricky, because first authors (or the core team leading the paper) do not want 50 people commenting in every decision, but they don’t want coauthors to end up not contributing much. On the other hand, some coauthors want to be more involved than others, but they need to be offered the opportunity to contribute in order to do so. I would recommend to fix at least three points to provide feed-back. First a draft of the questions, hypothesis to be tested and which approach will be used. Second a draft with the main results/ figures. And third a first draft of the paper. Even this seems really a minimum, I made myself the error of not sending almost anything before the complete draft of the paper was ready to some coauthors.

3. Make all correspondence open. Always include all coauthors in the emails with drafts or results to discuss. All coauthors should be able to see other people comments. This is specially important when two coauthors disagree on something. The lead author has the final word, but other coauthors should discuss the disagreement between them (and hopefully agree on something) and in front of all other coauthors.

4. Be clear on what you want. That applies also to both sides. As a first author is very useful to tell people what do you want from them. Instead of letting people comment on whatever they want (they will do that anyway), ask for specific questions. Can some native speaker check my grammar? Can you go trough the mathy part and make sure it is correct? With several coauthors you have the risk that everyone will hope the other will look on the three pages of equations, and no-one ends up doing it. As a co-author is always nice also to state on what do you want to contribute. Even if you think that you will be “near the end of the list”, if you want to be more engaged and have clear ideas on how to redo an analysis, or enhance a figure, say it! (Author order is /should be flexible, so you may end up among firsts authors if you contribute)

Lastly, those are just suggestions, and all of them refer to one basic idea: enhance communication.

*I do think more than 10 authors are rarely useful…

I have a guest post in Practical Management blog

Quick note to say I am very glad to have a guest post in an awesome blog about data management, a neglected topic that affect all scientists. The blog is quite funny also, bringing some glamour to the art of data processing. Thanks Christie for inviting me to contribute!

The post is about style, check it out here: http://practicaldatamanagement.wordpress.com/2014/02/24/guest-post-about-style/