© 2004 Foundation for Promotion of Cancer Research
Individual Patient-level and Study-level Meta-analysis for Investigating Modifiers of Treatment Effect
1 Division of Clinical Trial Design and Management, Translational Research Center, Kyoto University Hospital, 3 Department of Epidemiological and Clinical Research Information Management, Kyoto University Graduate School of Medicine, Kyoto and 2 Department of Biostatistics, School of Health Sciences and Nursing, University of Tokyo, Tokyo, Japan
For reprints and all correspondence: Satoshi Teramukai, Division of Clinical Trial Design and Management, Translational Research Center, Kyoto University Hospital, 54 Shogoin Kawara-cho, Sakyo-ku, Kyoto, 606-8507, Japan. E-mail: steramu{at}kuhp.kyoto-u.ac.jp
Received May 31, 2004; accepted October 6, 2004
 |
Abstract |
|---|
 TOP Abstract INTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Background: In meta-analyses of clinical trials, clinicians are often interested in examining subset effects. Meta-regression of aggregated data is a usual approach for relating sources of variation in treatment effects to specific study characteristics. However, it is known that study-level analyses can lead to biased assessments and have some limitations in explaining the heterogeneity. An individual patient data (IPD) meta-analysis offers several advantages for this purpose.
Methods: We compared some regression analyses of IPD with meta-regression analyses of the summarized data using a real-world example in order to investigate whether a binary patient characteristic is related to treatment effect. We used data from 10 randomized trials for non-small-cell lung cancer (n = 1355).
Results: For treatment x stage interaction in IPD regression analysis, none of the tests of interactions was statistically significant. The meta-regression analysis gave a greater P-value than the IPD analysis. When excluding two studies, which had only stage I patients, the interaction was also not statistically significant in IPD analysis. On the other hand, the result of meta-regression analysis, though also showing no significant relationship, revealed a clear reversal in the direction of effect.
Conclusion: We suggest that the results of meta-regression analyses would not be as robust as those of regression analyses using IPD in examining potential modifiers of treatment effects. To investigate whether patient characteristics are related to treatment effects, we suggest that interaction tests and sensitivity analyses using IPD should be employed whenever possible.
Key Words: meta-analysis • heterogeneity • meta-regression • individual patient data • interaction test
|
INTRODUCTION |
|---|
|
TOPAbstractINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Meta-analysis is a tool in the continual process of clinical research for the discovery of new knowledge and also a scientific basis for planning future research. Assessment of between-study heterogeneity is one of the prime values of meta-analysis from this standpoint (1,2). Although some statistical models such as random-effects models account for heterogeneity to estimate a summary effect measure, they do not provide a method of exploring the reasons why study results vary. Heterogeneity between results from various clinical studies may occur, for instance, when patient populations vary across them or when patient characteristics are related to treatment effects.
Meta-regression of aggregated data is the usual approach in relating sources of variation in treatment effects to specific study characteristics (3). Meta-regression is a regression analysis in which the study estimates of treatment effect are the response variables, while study-level covariates, each of which has a value defined for each study, are the explanatory variables. For example, meta-regression analysis was undertaken to explore differences in graft-versus-host disease (GVHD) incidence between peripheral blood stem cell transplantation (PBSCT) and bone marrow transplantation (BMT) (4). In a cancer epidemiology study, the effects of alcohol and tobacco on the upper aerodigestive cancers were examined using the meta-regression technique (5). However, it is known that study-level analyses can lead to biased assessments, and use of aggregated summary values has some limitations for explaining the heterogeneity (6–8). An individual patient data (IPD) meta-analysis offers several advantages for this purpose. Our goal is to compare some regression analyses of IPD with meta-regression analyses of the summarized data using a real-world example, in order to investigate whether a binary patient characteristic is related to treatment effect. We specifically address the circumstance when an entire study may be homogeneous with respect to a level of a categorical covariate, i.e. the study includes patients only in one category of a categorical covariate.
|
METHODS |
|---|
|
TOPAbstractINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
DATA
Using data from the randomized trials for investigating the efficacy of adjuvant immunochemotherapy (a streptococcal preparation, OK-432) after resection of non-small-cell lung cancer, we sought to examine whether the stage of diseases, which was categorized into stage I or stage II–IV, was related to treatment effect. The patient data were collected from 10 randomized trials from the meta-analysis reported by Sakamoto et al. (9). The patient data of study no. 5 were not available because the trial was conducted in Yugoslavia. The size of the analysis set and the number of events in the present analysis were slightly different from the reported literature-based meta-analysis because of the availability of data and timing of the analysis. In these trials, the primary end-point was overall survival and the median follow-up time was 4.9 years.
The summarized results and characteristics of the studies are shown in Table 1, using a hazard ratio of death as a summary of the results in each study. Four patients in study no. 8 with missing values for the stage of disease were excluded. A total of 1355 subjects (treatment group, 686; control group, 669) were included in the present analysis. Two studies (study nos 2 and 10) had patients exclusively with stage I.
|
The overall effects were estimated using the Cox proportional hazards model stratified by the study (10). The overall treatment effect was statistically significant [hazard ratio 0.80, 95% confidence interval (CI) 0.69–0.93]. The test for heterogeneity in the log-hazard ratio across studies yielded
2(9) = 14.9 (P = 0.094), and this indicated that the null hypothesis of no heterogeneity in the log-hazard ratio cannot be rejected at the 0.05 level of significance. In subgroup analysis according to the stage of disease, there was a statistically significant treatment effect among patients with stage I (n = 740; hazard ratio 0.71; 95% CI 0.55–0.92). On the other hand, no significant treatment effect was observed in patients with stage II–IV (n = 615; hazard ratio 0.86; 95% CI 0.71–1.04).
INDIVIDUAL PATIENT-LEVEL ANALYSIS
To evaluate treatment x stage interaction in the IPD regression analysis, we considered two types of regression models. Let subject i be a member of study j, j = 1, ..., k (k = 10), which follows a stratified Cox regression model (10). The hazard function for subject i in study j can be written as
 |
0j(t) is the baseline hazard for subjects in study j, treatij is coded 0 for the control group and 1 for the treatment group, stageij is coded 0 for stage I and 1 for stage II–IV, and (treat x stage)ij is the treatment by stage interaction. Their corresponding regression coefficients are ß1, ß2 and ß3. In model 1, eß1 means the hazard ratio for treatment effect in stage I, e(ß1+ß3) means the hazard ratio for treatment effect in stage II–IV, and ß3 = 0 means no difference in treatment effect between stages. The analysis based on model 1 is the most common approach for carrying out IPD meta-analysis for survival data. Thus, model 1 is regarded as a reference model in terms of the validity of the results.
To compare with the meta-regression analyses using summary data, which had risk ratio as its measure of effect, we changed the response variable from survival time to a binary outcome, i.e. alive or dead at the end of study, because information on the number of those alive or dead at a fixed time might be the only data available in any selected literature for meta-analysis. We used exponential risk models to examine interactions between the treatment and the stage of disease. The fixed-effects model can be written as
 |
STUDY-LEVEL ANALYSIS
In the study-level analysis, we assumed that the only information available is the number alive or dead at the end of study and aggregated covariate data. The meta-regression model using summary data can be written as:
 |
j where j is the variance of the log (RRj) (11). All analyses were performed using SAS version 8.2 (SAS Institute Inc., Cary, NC). |
RESULTS |
|---|
|
TOPAbstractINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
COMPARISON OF METHODS
The results of the IPD analyses are shown in Table 2. In model 1, the estimated hazard ratio for treatment effect in stage I (the estimates for eß1) was 0.72, the estimated hazard ratio for treatment effect in stage II–IV (the estimates for e(ß+ß3)) was 0.85, and the estimated interaction term (ß3) was 0.169. In model 2, the estimated risk ratio for treatment effect in stage I (the estimates for eß1) was 0.77, the estimated risk ratio for treatment effect in stage II–IV [the estimates for e(ß1+ß3)] was 0.91, and the estimated interaction term (ß3) was 0.172. In the regression models using IPD, none of the tests of treatment x stage interactions was statistically significant. These results indicated that the treatment effect might not be different between stage I patients and stage II–IV patients.
|
Figure 1 shows a bubble plot of the log-risk ratio against the proportion of patients with stage II–IV. A bubble shows a study and the size of bubble is proportional to the inverse of the variance of the log-risk ratio. The meta-regression analysis using summary data gave a greater P-value than model 2, which are models for log-risk ratio (slope, 0.20; standard error of the slope, 0.35; P = 0.568).
|
SENSITIVITY ANALYSIS
When excluding two studies (study nos 2 and 10), both of which were only on stage I patients, we found that the interaction term was also not statistically significant in any models (Table 3). The P-value for interaction tests was larger than the overall results. The estimated treatment effects in stage I patients (the estimates for eß1) were 0.78 and 0.82 in model 1 and model 2, respectively. They were slightly smaller after removing the two studies than before removing them. In stage II–IV patients, the estimates of the treatment effect (the estimates for e(ß1+ß3)) were 0.85 and 0.91 in model 1 and model 2, respectively, and they remain unchanged.
|
On the other hand, the direction of the effect from the meta-regression analysis was dramatically altered (slope, –0.86; standard error of the slope, 0.70; P = 0.218) with no significant relationship between the treatment effect and the stage (Fig. 2). The point estimate of the slope was changed from positive to negative after excluding two studies.
|
|
DISCUSSION |
|---|
|
TOPAbstractINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Investigating potential sources of heterogeneity is an important component of carrying out a meta-analysis. The underlying level of risk can surface as a key variable related to a given treatment effect (12). In the present study, we focused on the stage of disease as a candidate of a treatment effect modifier because disease stage is one of the most important prognostic factors in any cancer. In the conventional subgroup analysis according to the stage, there was a statistically significant treatment effect among patients with stage I. On the other hand, there was no significant effect in the patients with stage II–IV. Some reports emphasize that an inappropriate subgroup analysis may lead to an incorrect conclusion (13,14). They recommended that statistical tests of interaction should be used rather than inspection of subgroup P-values. Therefore, we performed the interaction tests using Cox regression models and exponential risk models.
In cases where no individual patient data are available, meta-regression of summary data has been used to investigate heterogeneity of treatment effects. For example, Cutler et al. showed a fitted equation for the relationship between the relative risk of GVHD and the difference in number of T cells; however, the relationship did not reach statistical significance (4). In the meta-regression analysis, only five studies were included and the sample size per study ranged from 37 to 350. We also showed using an example that study-level meta-analysis has lower power than an equivalent IPD analysis. With regard to the statistical power for investigating heterogeneity of treatment effects in various situations, Lambert et al. performed a simulation study for meta-analyses including five, 10 and 20 studies, each of 200, 500 and 1000 patients (6). In the case of 10 studies of size 200 with a small effect size, 21% of the IPD meta-analyses are significant at the 5% level, while only 7% are significant for the meta-regression. They showed that the IPD analyses have greater power for any of the simulated situations. The intra-class correlations are consistently low (range 0.06–0.22), indicating that there is little agreement between the two methods. Even if appropriate statistical methods have been used for meta-regression, there are a number of limitations to the interpretation of the results (8,15).
As we have shown in the present study, for categorical covariates, one of the limitations is potential confounding across studies. In our example, two studies (study nos 2 and 10) have no information concerning the difference of treatment effects between stage I and stage II–IV, because these studies only have stage I patients. The result of meta-regression were changed if we use all the information (11 studies), or if we use only information within studies (nine studies). Therefore, we should always interpret the results while considering this kind of confounding between covariates and studies. Another major drawback of meta-regression is aggregation bias or ecological fallacy (11,16). The fallacy is the mistaken assumption that a statistical association observed between two group-level variables is equal to the association between the corresponding variables at the individual level. If we have no individual patient data, analysis using only published average data could be difficult to interpret, because a between-study relationship based on aggregated data might not reflect a within-study relationship based on IPD.
In conclusion, we suggest that the results of meta-regression analyses using summary data would not be as robust as that of regression analyses using IPD to examine potential modifiers of treatment effect. To investigate whether patient characteristics are related to treatment effects, we suggest that interaction tests and sensitivity analyses using IPD should be employed whenever possible.
|
Acknowledgments |
|---|
This paper was presented in part at the 1st US–Japan Biostatistics Workshop, Kobe, September 26, 2003. We would like to thank the Japanese Meta-Analysis Group in Cancer of the Japanese Society of Strategies for Cancer Research and Therapy, and Chugai Pharmaceutical Co. Ltd for permission to use the valuable data.
|
References |
|---|
|
TOPAbstractINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
1 Thompson SG. Why and how sources of heterogeneity should be investigated. In: Egger M, Davey Smith G, Altman DG, editors. Systematic Reviews in Health Care. London: BMJ Publishing Group 2001;157–75.
2 Schmid CH. Exploring heterogeneity in randomized trials via meta-analysis. Drug Inf J 1999;33:211–24.
3 Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med 1999;18:2693–708.[CrossRef][ISI][Medline]
4 Cutler C, Giri S, Jeyapalan S, Paniagua D, Viswanathan A, Antin JH. Acute and chronic graft-versus-host disease after allogeneic peripheral-blood stem-cell and bone marrow transplantation: a meta-analysis. J Clin Oncol 2001;19:3685–91.
5 Zeka A, Gore R, Kriebel D. Effects of alcohol and tobacco on aerodigestive cancer risks: a meta-regression analysis. Cancer Causes Control 2003;14:897–906.[CrossRef][ISI][Medline]
6 Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol 2002;55:86–94.[CrossRef][ISI][Medline]
7 Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI. Individual patient- versus group-level data meta-regressions for the investigation of treatment effects modifiers: ecological bias rears its ugly head. Stat Med 2002;21:371–87.[CrossRef][ISI][Medline]
8 Higgins JPT, Thompson SG. Controlling the risk of spurious findings from meta-regression. Stat Med 2004;23:1663–82.[CrossRef][ISI][Medline]
9 Sakamoto J, Teramukai S, Watanabe Y, Hayata Y, Okayasu T, Nakazato H, et al. Meta-analysis of adjuvant immunochemotherapy using OK-432 in patients with resected non-small-cell lung cancer. J Immunother 2001;24:250–6.[CrossRef][ISI][Medline]
10 Kalbfleisch JD, Prentice RL. The Statistical Analysis of Failure Time Data, 2nd edn. New York: Wiley 2002;118–9.
11 Greenland S. Quantitative methods in the review of epidemiologic literature. Epidemiol Rev 1987;9:1–30.
12 Davey Smith G, Egger M. Going beyond the grand mean: subgroup analysis in meta-analysis of randomised trials. In: Egger M, Davey Smith G, Altman G, editors. Systematic Reviews in Health Care. London: BMJ Publishing Group 2001;143–56.
13 Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000;355:1064–9.[CrossRef][ISI][Medline]
14 Brookes ST, Whitley E, Peters TJ, Mulheran PA, Egger M, Davey Smith G. Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. Health Technol Assess 2001;5:33.
15 Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med 2002;21:1559–73.[CrossRef][ISI][Medline]
16 Morgenstern H. Uses of ecological analysis in epidemiological research. Am J Public Health 1982;72:1336–44.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||