Methods and Standards

The Aggregation Challenge

We have multiple projects innovating on aspects of what we call the "aggregation challenge." The aggregation challenge is the challenge of combining findings from multiple studies to foster cumulative learning. We think of it as one of the biggest challenges facing social scientists and that addressing it will be key for strengthening the relevance and reliability of social science findings. 

  • Coordinated trials. IPI has contributed to the "metaketa" initiative housed at EGAP.  Metaketas are coordinated randomized trials across multiple sites with harmonized measures and analysis strategies. See this summary paper in Science Advances summarizing the results from Metaketa 1 and this "shiny app" we developed that lets readers explore analyses and robustness of findings to the inclusion or exclusion of  different studies.
  • Coordinated analysis of strategies to measures hidden populations. We are leading the meta-analysis for a multicountry collection of studies coordinated by  APRIES  to assess the prevalence of human trafficking prevalence estimation. See our hiddenmeta package at https://gsyunyaev.com/hiddenmeta/ 
  • Meta-models: Our Correlates of Corona project examines socioeconomic predictors of Covid mortality. Experimental stages now in the field focus on aggregating disciplinary beliefs about logics driving Corona and strategies to connect observational patterns with causal logics. See here for our challenge.

Related writing:

Qualitative and Mixed Methods Inference 

We have been developing approaches to better integrate qualitative and quantitative strategies and better integrate theoretical and empirical research. The core idea is to  represent theories as causal models, use Bayesian models to update on these theories given some mixture of 'correlational' and 'process' data, and then query those models to infer causal relations for specific cases of interest. 

  • In October 2023, Humphreys published Integrated Inferences together with Alan Jacobs with Cambridge University Press. Integrated Inferences provides strategies for using causal models to integrate inferences from qualitative and quantitative data and to provide better theoretical insights into qualitative work. The key strategy is to use Bayesian approaches to update simultaneously from cross-sectional information on causes and outcomes and process information about the operation of causal mechanisms. For more see integrated-inferences.github.io
  • We maintain the CausalQueries R package that lets users define and combine causal models. See our guide for examples with applications to combine inferences from qualitative and quantitative analyses, inferences from observational and experimental studies, and inferences from multiple trials examining different parts of a common causal model. See introductory slides. Sisi Huang developed a shiny app to allow easy access to core functionality.

Research Design

The 'credibility revolution' has led to a greater focus on the quality of research designs when assessing what inferences to draw from published research. Surprisingly however there are few common standards for either characterizing designs or assessing their quality.      

  • In August 2023, Humphreys, together with Graeme Blair and Alex Coppock, published Research Design in the Social Sciences with Princeton University Press. This book provides a framework to characterize and improve research designs and enhance research transparency. The framework has been integrated into the EGAP research design registry, promoted on the World Bank development blog, and is being tested in multiple research labs.  With agreement of Princeton University Press the book is available open access.
  • A “shiny” interface for the framework was developed at WZB by Lily Medina, Clara Bicalho, and Markus Konrad. 
  • Power analysis with DeclareDesign

Doing No Harm When Conducting Research in Development Countries, What about the researchers?

The purpose of this project is twofold: First, it identifies and synthesizes evidence of ethical challenges that are faced by local and international research staff implementing field research projects in the Global South. Second, the project aims at critically assessing and reviewing existing ethical guidelines and protocols that seek to address and alleviate these challenges. This will then serve as a guidance for developing normative ethical principles and standardized research guidelines that take into account a) the specific complexities linked to research in developing country contexts and b) the protection of local and international research staff at all hierarchical levels.

Researchers:  Ana Garcia-Hernandez (WZB), Lennart Kaplan (DIE), Jana Kuhnt (DIE), and Janina Isabel Steinert (TUM)

Status: Link to project page