P-hacking, or researcher degrees of freedom, it’s a worrying issue in science. Specially, because p-hacking is not a black and white issue. On the blackest side there is deliberate p-hacking with the only purpose to advance your career. This is bad, but I hope it’s rare*. The grey area is more intriguing, because it concerns researchers not doing it consciously. I used to think that this include researchers that never had a proper statistical training, with too much pressure to publish too small datasets or that fool themselves thinking that this new analysis/subset of the data is what he/she should be testing in first place, so it doesn’t matter really the 200 previous analysis/subsets (which is false, they matter!). This is equally bad for science (even if the motivation is not as bad).

But then I read “against method” of Paul Feyerabend**. Despite some passages are really slow and repetitive, I liked it. A big part of the book explains Galileo Gallilei story. Galileo changed the paradigm based in incomplete theory, iffy data and measurement tools, and lots of propaganda. He used more its intuition than a proper scientific method. He was still right and most of his ideas were confirmed years later.

And that rang a bell. I’ve heard before scientists saying things like “well, we can’t measure it accurately, but trust me I know the system and this is what is happening”. From here to do a bit of conscious or unconscious p-hacking to support your hypothesis there is a small step. This researchers are using intuition, hours of thought and lots of knowledge. This scientists are putting forward their ideas. Ideas in which they believe, but they can’t just prove unequivocally with the data at hand because of the complexity of the problem.

Paul Feyerabend said that “everything goes” if it advances science. I am not justifying p-hacking to support something that it’s hard to  prove but you think is true, but after reading Feyerabend I am also less worried about adding some subjectivity to the scientific method, because being completely objective and following the method strictly may also slow down science. Maybe the middle ground is being able to recognizing when something is an opinion, and not facts, and avoid sticking a p-value to this opinion, but defend it anyway in the light of the data available and try to push forward the agenda to get better data, better methods, or whatever you need to support it. It’s complicated.

*people that only want to advance their careers choosed politics in first place, not science, right?  I know this is probably a wrong assumption.

**In a nutshell he praises that an objective scientific method is unattainable and rarely applied, and that we should free ourselves from using it as the single tool to do science. I liked for example the idea of aiming to create a plethora of theories (with no historical constraints or resistance from the status quo to accept compatible alternative explanations) that can cohabit and let time to do the thinning a posteriori. More on wikipedia.