Sample size matters – even more than you think

Posted on 20th May 2013 by Alice Buchan

A large sample size is considered a plus point in a study. Why?  It increases statistical power – that it reduces your chance of a Type II error (false negative) [1]. Large sample sizes also allow you to account for data that may be extremely variable [1]. A recent meta-epidemiological analysis of systematic reviews suggests that the sample size may also influence the effect size – which must be taken into account when reading the literature [2].

The authors looked at a total of 93 meta-analyses, which included more than 700 trials. These analyses covered a wide range of diseases; depending on how common the disease is, 1000 people may be a very small trial (e.g. heart disease), or for a less common condition may be very large. Thus, the authors ranked the trials included in the meta-analysis by size, and compared all trials in the largest quarter (of their individual meta-analysis) to the other three quarters- and found that the smallest trials (in the lowest quarter of their meta-analysis) found the largest effect size. (Fig 1 in the paper). They also decided to look at the trials ranked by size overall, and found a similar pattern (second panel of Fig. 1).

Odds ratio for effect size compared to sample size

This poses some interesting questions, since effect size is an important consideration when considering whether or not to implement a clinical intervention. The authors propose that one of the reasons for this effect may be a greater publication bias (towards those trials with positive results) in smaller trials, whereas very large trials may be more likely to be published regardless of the outcome. They have also suggested an outcome reporting bias in smaller trials. An alternative possibility is that larger studies have greater heterogeneity, and this may lead to a smaller effect size. Smaller trials are also not powered to identify smaller effect sizes [1]. As this analysis shows, it is important to consider the sample size of a study when interpreting the results and considering how the information gained should be applied.

References

[1] Biau DJ, Kernéis S, Porcher R. Statistics in Brief: The Importance of Sample Size in the Planning and Interpretation of Medical Research Clin Orthop Relat Res. 2008;466:2282-8

[2] Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013;346:f2304

Tags: