The Campbell Collaboration Colloquium location:Oslo date:18-20 May 2009
In a meta-analysis, the findings of a set of studies are combined and compared in a quantitative and systematic way. Since in primary studies typically not the whole population of interest, but only a sample is included, a plausible reason why findings vary over studies is sampling error. In addition there is often some systematic variation between study results since studies frequently differ from each other in, for instance, the kind of participants, the treatment and the design. Multilevel models are increasingly used to model both the between-study and the sampling variation, and look for moderator variables to explain the between-study variation. A major advantage of using multilevel models for meta-analysis is their amazing flexibility, allowing fitting models that may better match the kind of data and the research questions. One possibility that is seldom mentioned in the methodological meta-analytic literature or typically is not implemented in software for meta-analysis, is the distinction of a third level of variation to model dependencies between studies (e.g., occurring when several studies stem from the same research group) or within studies (e.g., occurring when within a study multiple samples were drawn). We will present some examples of three-level meta-analyses we performed and explore the required number of units at each of the levels to achieve enough power and accuracy in testing and estimating moderator effects and variance parameters. To this end, we will make use of ML-DEs, a program we recently developed to explore the design efficiency for a variety of multilevel models.