Berkeley Initiative for Transparency in the Social Sciences

Home » News » Scientific Irreproducibility and the Prospects of Meta-Research

Scientific Irreproducibility and the Prospects of Meta-Research

recent article from The Economist featuring John Ioannidis’ Meta-Research Innovation Center (METRICS), whose work to advance the credibility of research will be presented next week at the BITSS Annual Meeting.


“Why most published research findings are false” is not, as the title of an academic paper, likely to win friends in the ivory tower. But it has certainly influenced people (including journalists at The Economist). The paper it introduced was published in 2005 by John Ioannidis, an epidemiologist who was then at the University of Ioannina, in Greece, and is now at Stanford. It exposed the ways, most notably the overinterpreting of statistical significance in studies with small sample sizes, that scientific findings can end up being irreproducible—or, as a layman might put it, wrong […] Dr Ioannidis has been waging war on sloppy science ever since, helping to develop a discipline called meta-research (ie, research about research).

METRICS’ mission is to “identify and minimize persistent threats to medical-research quality.” These include irreproducibility of research findings (the inability of external researchers to reproduce someone else’s work, most often because research data is not shared or data manipulations are not correctly detailed), funding inefficiencies (supporting flawed research), and publication bias (not all studies that are conducted get published, and the ones which do tend to be show significant results, leaving a skewed impression of the evidence).

The fields of medicine and public health have done tremendous progress addressing these issues. Some social science disciplines are catching up, as Psychology’s Many Labs Replication Project testifies. But the norms and incentives that govern the social sciences make certain challenges specific to its disciplines. What can social science learn from medicine’s reproducibility crisis? That’s the main question to be addressed by John Ioannidis next week at the BITSS Annual Meeting. You can register here.


1 Comment

  1. Maria Saraiva says:

    Besides statistical part, which is a huge problem, there seems to have a major problem that has not been addressed. With this idea of massive paper production, graduate programs have focused on publication and not to form scientist. Student’s and their advisors are not interested in studying because they need to have at the end of PhD at least 10 or 15 papers. How many of those with PhD titles really know what empirical science is? I have been asking many famous researchers who has more than 30 papers a year in excellent journals and they have no idea of what empirical is. So they have no grasp of what they are doing. Moreover, for most of empirical research we need statistical knowledge. However, again, ask these marvelous, best paper producers in the best Journals of the world if they really can make interpretations from the paper they read to fundament their research? The answer is they have no idea, and no time to study they need to produce papers. I already talked with many people from medical area planning case-control studies, and they don’t even bother about control selections which is always from hospitals because if not it is impossible for them and they don’t care. People are graduating with PhD in Epidemiology in certain famous places of the world with no idea of what sampling design is for, they say surveys are like Impressionist Paint, you have to see the data from far so you can realize what the data is telling you, even if this data does not represent the population”. These comments are found from pretty important public health leaders. Let’s say, without time to study to become a good pianist we never will get there. But scientists are born in bundles without no need to study. I am glad I graduate from Michigan where I did have time to study and good examples from my teachers. My PhD was to learn and show I had learned not to produce 10 papers and I am glad it was that way. However, a lot of people say it was waist of time, instead I should have just published. However, how I would teach biostatistics and epidemiology without really knowing it? I would´t as many dont, and then after some years I would say, “I had no idea in 2003 that survey desing was importante”. I think things will just get worst if we dont change our PhD programs. How a prolific paper Publisher has no idea of what empirical are? So they have no idea of limitations and the context of what they do. Actually Science for them is to produce whatever dada and publish it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

%d bloggers like this: