Berkeley Initiative for Transparency in the Social Sciences

Home » Events & Workshops » Research Transparency Meeting with CGD

Research Transparency Meeting with CGD

By Garret Christensen (BITSS)


Though BITSS hopes to increase research transparency across the social sciences, several of us, myself included, have a background in development economics. So we were happy to take part in a meeting last week at the Center for Global Development (CGD) in Washington, DC. In addition to BITSS and CGD, representatives from the International Initiative for Impact Evaluation (3ie), Inter-American Development Bank, InterAction, Innovations for Poverty Action Lab (IPA), Millennium Challenge Corporation (MCC), World Bank research group, United States Agency for International Development (USAID), and the US Treasury were present.

I was impressed by how much agreement there was, and how interested these large, sometimes slow-moving, organizations seemed to be, but I should probably temper my enthusiasm a bit: the people in the room were not randomly selected from their respective agencies, and even if they had been, we may still be far from actual policy changes and wider adoption. Regardless, we had a fruitful discussion about some of the roadblocks on the way to increased transparency.

Here are a few of the themes we discussed, mostly obstacles to increased transparency:

  1. Bill Savedoff and Michael Clemens from CGD suggested we need to increase the demand for transparency if we expect people to supply it. So if we want people to share their data, or to write exhaustive appendices, original surveys, and protocols so that others can reproduce their work, we should actively be asking for, reading, using, or citing the protocols that are out there. Others suggested we needed to change norms so that transparency becomes the default for how research is done. I was a little surprised that in a room with so many economists, I actually had to be the one to say that money could pretty easily create norms: if funders held back the last 10% of their grant until data and code was online, then boom, we’re done, data gets shared.
  2. Justin Sandefur and David McKenzie suggested that adoption of state-of-the-art methods is fairly common among the RCT crowd, but there’s been less buy in from observational researchers. Justin also suggested we should decrease the length of time authors are granted a monopoly on their collected data.
  3. Representatives from big development project implementers such as the World Bank and the Millennium Challenge Corporation have a hard time actually defining what their interventions are, let alone evaluating them. How can you pre-specify a hypothesis if you’re constantly re-evaluating what the intervention is or should be? The phrase “a project in search of a plan” was used to describe the process of identifying an intervention and a evaluation design.
  4. Replication to scientists is like taxes to liberals, said 3ie’s Annette Brown. Great in theory, but President Obama’s brief interest in taxing 529 education plans shows that getting approval in reality is often much more difficult. 3ie has been running a replication project and it has faced a few issues: low interest from researchers and publishers, longer than expected duration, and maybe some researchers unhappy with the process. Given that it’s something so many researchers say they want, what is the right role for a 3rd party to play so that replications become more prevalent?

If you have any suggestions on solutions to these issues, we’d love to hear them! Attendees expressed interest in a larger meeting to continue discussion and bring more funders and implementers to the table, possibly in the fall. We’ll keep you posted.


2 Comments

  1. Thanks Garret! That’s not quite what I meant by the reference to the 529 plans. I was instead admitting that while I argue for the importance of taxes generally, I was very much against this particular tax that would have directly affected me. We have found that replication for researchers is like taxes for liberals in that way–a great thing until it happens to you.

  2. Yep, I got that. Sorry if I was unclear. Unfortunately a lot of things are like this, including the mortgage interest deduction, which is quite easy for me to say as a renter.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

%d bloggers like this: