Heterogeneity: what is it and why does it matter?

Posted on 29th November 2018 by Maximilian Siebert

Heterogeneity is not something to be afraid of, it just means that there is variability in your data. So, if one brings together different studies for analysing them or doing a meta-analysis, it is clear that there will be differences found. The opposite of heterogeneity is homogeneity meaning that all studies show the same effect.

It is important to note that there are different types of heterogeneity:

• Clinical: Differences in participants, interventions or outcomes
• Methodological: Differences in study design, risk of bias
• Statistical: Variation in intervention effects or results

We are interested in these differences because they can indicate that our intervention may not be working in the same way every time it’s used. By investigating these differences, you can reach a much greater understanding of what factors influence the intervention, and what result you can expect next time the intervention is implemented.

Although clinical and methodological heterogeneity are important, this blog will be focusing on statistical heterogeneity.

How to identify and measure heterogeneity

Eyeball test

In your forest plot, have a look at overlapping confidence intervals, rather than on which side your effect estimates are. Whether the results are on either side of the line of no effect may not affect your assessment of whether heterogeneity is present, but it may influence your assessment of whether the heterogeneity matters.

With this in mind, take a look at the graph below and decide which plot is more homogeneous.

Of course, the more homogeneous one is the plot number 1 . The confidence intervals are all overlapping and in addition to that, all studies favour the control intervention.

For the people who love to measure things instead of just eyeballing them, don’t worry, there are still some statistical methods to help you seize the concept of heterogeneity.

Chi-squared (χ²) test

This test assumes the null hypothesis that all the studies are homogeneous, or that each study is measuring an identical effect, and gives us a p-value to test this hypothesis. If the p-value of the test is low we can reject the hypothesis and heterogeneity is present.

Because the test is often not sensitive enough and the wrong exclusion of heterogeneity happens quickly, a lot of scientists use a p-value of < 0.1 instead of < 0.05 as the cut-off.

I²

This test was developed by Professor Julian Higgins and has a theory to measure the extent of heterogeneity rather than stating if it is present or not.

Thresholds for the interpretation of I² can be misleading, since the importance of inconsistency depends on several factors. A rough guide to interpretation is as follows:

• 0% to 40%: might not be important
• 30% to 60%: moderate heterogeneity
• 50% to 90%: substantial heterogeneity
• 75% to 100%: considerable heterogeneity

To understand the theory above have a look at the following example.

We can see that the p-value of the chi-squared test is 0.11, confirming the null hypothesis and thus suggesting homogeneity. However, by looking at the interventions we can already see some heterogeneity in the results. Furthermore, the I² Value is 51% suggesting moderate to substantial heterogeneity.

This is a good example of how the χ² test can be misleading when there are only a few studies in the meta-analysis.

How to deal with heterogeneity?

Once you have detected variability in your results you need to deal with it. Here are some steps on how you can treat this issue:

• Check your data for mistakes – Go back and see if you maybe typed in something wrong
• Don’t do a meta-analysis if heterogeneity is too high – Not every systematic review needs a meta-analysis
• Explore heterogeneity – This can be done by subgroup analysis or meta-regression
• Perform a random effects meta-analysis – Bear in mind that this approach is for heterogeneity that cannot be explained because it’s due to chance
• Changing the effect measures – Let’s say you use the Risk Difference and have high heterogeneity, then try out Risk Ratio or Odds Ratio

References

(1) Fletcher, J. What is heterogeneity and is it important? BMJ 2007; 334 :94

(2) Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

(3) https://www.mathsisfun.com/data/chi-square-test.html

Tags: