Three Reasons You Need to be Wary of Subgroup Results in Studies
Intermediate Data Literacy
“Fenofibrate may reduce the risk of heart disease in some patients with type 2 diabetes.”
If you have diabetes, you might find a story like this tempting. It seems to promise cardiovascular benefits from taking cholesterol-controlling drugs.
It sounds like it might be time for you to ask them about the benefits of fenofibrate, right? Considering diabetes is a risk factor for heart disease.
However, the positive results that show a benefit came from a smaller subset of people within the study. Researchers can use a statistical method dubbed a “subgroup analysis” to look at the effect of an intervention, like a drug, on a small group of people.
Looking more closely at the data, it was found that people with certain triglyceride levels and HDL cholesterol had a positive effect of the drug. But if we conduct a bigger study with fenofibrate, which was done, we might find no decreased risk.
This is why we need to be careful when interpreting parts of studies like this, again called a subgroup analysis. These subgroup analyses can be a great tool for generating hypotheses, but they can’t definitively answer the question of whether or not treatments work.
Here’s why:
1. Subgroup results are more likely to be skewed
Randomization is a very powerful tool against bias in research. It is done by assigning study participants to the control or treatment group based on their chance.
The assignment is completely random, which saves the process of human susceptibility to making biased decisions.
It's important to keep your study samples as balanced as possible and account for potential confounds by ensuring that different groups are similar with regard to key patient characteristics.
However, sometimes there are differences within groups that matter and are of interest to researchers to look at.
But there are some risks with subgroups. If a group is small, people may not be completely representative of the population and an imbalance can create skewed results.
Consider what might happen if there were more patients who are older and sicker are on a placebo in a study, than those receiving the active treatment. The people on the placebo arm might seem to be doing better, just because they are healthier overall. Despite the apparent benefits of the placebo arm, there is a question as to how real these supposed benefits are.
It would be impossible to say for certain, but here’s something we can say with confidence: Subgroups have a lousy track record of identifying legitimate health benefits. A recent analysis of clinical trials found that only 5 out of 46 claims were tested in follow-up studies. But the groups that were tested for follow-up did not produce reproducible results.
“A minority of subgroup claims made in the abstracts of RCTs are supported by their own data,” the analysis concluded.
In other words, the findings rarely get tested on their own, and when they do, the results are lackluster.
2. Positive subgroup results are more likely to be due to chance
Another issue with subgroups is that they are more likely to produce low power results, that seem statistically significant but are really just reflect chance. The lower the sample size, the higher your false-positive rate will be
Studies often make multiple comparisons, which raises the risk of false positives. For instance, a study with 10 subgroups carries a 40% risk of a false positive "hit" assuming there’s typically a 5% chance that you’ll randomly hit on a false-positive.
It is also problematic that so few researchers specify their research plans before they conduct a study. "Post-hoc" analyses can be a problem in research if the researcher is not careful enough. This can result in false-positive findings classically seen from p-hacking and data dredging.
It's always good to have some type of specification related to subgroups in your research, but the findings still should be speculative until stronger research is conducted.
3. Subgroups have reduced statistical power
Although not always the case, reports on scientific studies often show only positive benefits from a certain treatment. This can be misleading in some cases when there are subgroups.
Subgroups are smaller parts of the main study and due to this, they have less power to identify subtle differences.
There are cases where inconclusive results can be misleading. This is the case when studies aren't able to measure the risk or benefits of specific groups, and is often encountered in situations with low response rates.
We see these low response rates in subgroup analyses.
Final Thoughts
Subgroup analyses are a statistical method to look at the effect of an intervention on a small group of people.
These analyses might be good for generating hypotheses for future studies, but they’re greatly limited. We outlined three reasons as to why:
Subgroup analyses are more likely to be skewed.
Positive findings are more likely due to chance.
Subgroups have reduced statistical power.