HTA submissions rely heavily on comparisons of treatments, and numerous methodologies have been developed to estimate relative treatment effects across studies in a rigorous fashion. The traditional methods (such as network meta-analyses) use study-level data to compare treatments connected through a common comparator. Population-adjusted methods, including matching-adjusted indirect comparison (MAIC) and simulated treatment comparison (STC), extend these approaches to account for differences in patient characteristics (prognostic and effect-modifying factors) across the studies included in the analyses. The choice of comparative method depends on the available evidence base, access to patient-level data, the structure of the treatment network, and the degree of heterogeneity between study populations.

Population-adjusted methods are particularly useful when the included studies have fundamentally different patients. Treatments are evaluated in separate studies that may differ in numerous ways, particularly with respect to patient population and baseline characteristics, making naïve cross-trial comparisons potentially biased. MAIC has become a popular population-adjusted indirect comparison method designed to address these challenges by reweighting individual patient data (IPD) from one study so that the weighted baseline characteristics match the aggregate characteristics reported in another study. This approach aims to reduce bias arising from differences in effect modifiers and prognostic variables across trials. STC is an alternative to MAIC, which also provides a method for population adjustment, and while not always a perfect solution, it may perform better in some challenging situations; however, to date, is a less-frequently used methodology

The MAIC methodology is used in situations where IPD are available for only one treatment comparison, while only published aggregate data are available for the comparator study. By assigning weights to patients in the IPD study, MAIC creates a synthetic population that more closely resembles the comparator study population, enabling a more balanced indirect comparison of treatment effects. The resulting estimates can provide decision makers with evidence to inform reimbursement, regulatory, and clinical decisions in the absence of direct comparative evidence.

Broadstreet has used MAICs successfully on a number of occasions, including to compare outcomes for different treatments in patients with TKI-naïve ROS1+ locally advanced or metastatic non-small-cell lung cancer, as well as to compare outcomes among treatments for migraine.  MAICs do, however, have limitations and are only useful in specific cases. It is the Goldilocks of comparative effectiveness research. The data have to be ‘just right’ before the MAIC can do what it purports to do. In cases where the two populations are too similar, the MAIC weightings are negligible, and the results match those of the unweighted analyses. And in cases where the populations are too different, the MAIC weights become unstable, and the results are unreliable (Figure). 

When baseline characteristics between studies are already highly similar, the impact of MAIC weighting may be modest. In these cases, the assigned patient weights tend to remain relatively uniform, preserving much of the original sample structure and maintaining a large effective sample size. The resulting adjusted estimates are often close to naïve indirect comparisons, suggesting limited cross-trial imbalance. Such scenarios generally increase confidence in the robustness of the comparison because there is substantial overlap between the trial populations and less reliance on extrapolation beyond the observed data.

In contrast, when studies differ substantially in important baseline characteristics, MAIC weighting can become more extreme. Large differences in prognostic factors or treatment effect modifiers may require assigning disproportionately high weights to a small subset of patients whose characteristics resemble those of the comparator population. This can markedly reduce the effective sample size and increase uncertainty in the treatment effect estimates. In severe cases, limited overlap between populations may indicate that the studies are not sufficiently comparable, raising concerns about the validity and interpretability of the adjusted comparison. These situations highlight the importance of evaluating population overlap and carefully considering whether MAIC is appropriate. 

How do we know if we are in the goldilocks zone? Are there ways to determine if the populations are ‘different enough’ to make the MAIC useful, but not so different that the methodology breaks down?  It is clear that being too similar is less of a problem than being too different. Populations that are already matched on all relevant prognostic and effect modifying characteristics are unlikely to produce biased estimates. The need for population adjustment becomes more pressing when there are differences across the populations, and more importantly, differences in the effect modifiers that have a strong effect on the outcomes.

Populations that show complete separation of relevant factors fall outside of the goldilocks zone—the MAIC methods cannot adjust for the effect of these factors. Results of such an MAIC are unreliable, and not relevant to the target population. Examples of this type of situation could include a study of exclusively adult patients that is being compared to an external study conducted on children. There are no weights that can be applied to adults that would produce an average equivalent to the age of the children. These populations are “too different” to benefit from the MAIC methodology. In actual practice however, it is not always easy to clearly identify when populations are too different. The effective sample size provides a good measure of the difference.

So, what can we do about this? If we are in the goldilocks zone, then we’re fine, but if we are outside of it we need to consider alternative methods. When the populations are too similar, there is little benefit to conducting and MAIC, but there is also little detriment (aside from wasted effort). The real dangers lie in the populations that are too different. STC (and other regression methods) are better at extrapolations outside the range of the data, so these can be used when one population contains covariate values that are outside the range of the other.

Broadstreet can advise on appropriate population-adjustment methods for treatment comparisons for HTA submission. Please get in touch and we will be happy to help!