- More data or better stats? Image via Wikipedia
What is going on? In the New Yorker we have Johan Lehner telling us that scientific findings are getting harder and harder to replicate. He worries that there might be something wrong with the scientific method. Meanwhile, on BBC4, Prof. Hans Rosling is revelling in the Joy of Stats and declaring this to be an unprecedented ‘Golden Age’ for science. A time when, thanks to the data deluge and computational super-crunching, we can see the world ‘as it really is’. So, who is right? Is scientific empiricism really in trouble and will correlation come to the rescue?
From the days of Francis Bacon onwards, replication was the gold standard for science. But are experiments now so complex and effects so subtle that noise will often swamp the real results and any significant findings are more a consequence of luck than judgment. Combined with a huge publication bias from editors and journals that aren’t interested all those dull null findings.
In our research methods classes we were all told time and again that ‘Mere correlations are what we were put on earth to rise above.’ But is that all we’re left with? John Ioannidis tells us that significance levels are spurious and misleading, and, as a result, ‘Why most published research findings are false.’ But build up a big enough dataset and then no-one can argue with what it says. Maybe?
“Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.
Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot”
Chris Anderson | Wired | The End of Theory: The Data Deluge Makes the Scientific Method Obsolete
Is this true and if it is true is it interesting? Because, of course, to some extent we do already do this. Several theories of personalities are wholly founded on factor analysis. Hans Rosling’s exciting stories of world health and global politics are built on solid data. Biology and medicine don’t just look at genes any more they look at genomes and genome-wide-associations. Your interpretation maybe different from mine but you can no longer go off to your lab and run a wholly unconnected experiment. You have to model my data too. All the data, not just the stuff that is sufficiently outlying to make it out of the desk drawer.
Science is less a sequence of individual elegant, experiments. It is a vast network of collected data. The answers are in there somewhere and statistics can still seek them out. The problems that Johan Lehrer has highlighted have come about not because science is broken but precisely because everyone is no more aware of the bigger picture.
The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.
Read more at NewYorker.com
Statistics is there to help us. It still provides scientists with an amazing bag of tricks, and amazingly those old favourites, averages and correlations are still effective. Science has to notice that just because the newest tricks use such old techniques, it doesn’t make them any less magical. Laboratory scientists need to overcome their prejudices against correlations and descriptive statistics.
This is not empirical anarchy, it is a data revolution.
Related articles
- Is Science Dead? In a Word: No (psychcentral.com)
- The Joy of Stats with Hans Rosling (flowingdata.com)
- Scientists Too Sure Of Themselves: Except, Of Course, Climatologists And Neuroscientists (wmbriggs.com)
- Is the “decline effect” really so mysterious? (scienceblogs.com)
Another potential response to the problems raised by Johan Lehrer comes from Bayesian statisticans. They’ve been arguing for ages that the classical statistics of Fisher and Student need to be replaced more robust and reliable Bayesian equivalents. I just found an article (not yet accepted for publication) that makes the point in an interesting and forceful way. Eric–Jan Wagenmakers, Ruud Wetzels, Denny Borsboom and Han van der Maas of the University of Amsterdam take a second look at some results that, on the face of it, are even more incredible. Daryl Bem has recently published a paper that appears to provide evidence for precognition, the ability to predict the future. If true it would be a truly revolutionary discovery. But exceptional claims require exceptional proof and it seems like Bem’s results don’t stand up to the harsh scrutiny of the Reverend Bayes. As Eric and colleagues expalin
Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
Eric–Jan Wagenmakers, Ruud Wetzels, Denny Borsboom and Han van der Maas
University of AmsterdamAbstract
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem’s data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.Full Article: www.ruudwetzels.com/articles/Wagenmakersetal_subm.pdf
.
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
