Clinical and Angiographic Risk Scores in Patients With ACS
Assessing the performance of prediction models is a complex process that requires the evaluation of several performance measures. Traditional measures for survival outcomes include discrimination, usually expressed as c statistic, and calibration, usually measured with the Hosmer-Lemeshow test. Discrimination is a measure of how well the prognostic model can separate cases from controls, whereas calibration is a measure of how well predicted probabilities fit actual risks. Typically, prognostic models cannot have both a perfect calibration and a perfect discrimination, and therefore, the best models will achieve an acceptable balance between these 2 measures. The c statistic, however, possesses inherent limitations because it is a function only of ranks and not of predicted probabilities, and therefore, it is insensitive to changes in absolute risk estimates. New measures have recently been proposed that offer incremental information over the c statistic. In this regard, the net reclassification improvement and the integrated discrimination index have recently been proposed as further measures to assess improvement in model performance. The present study is the first to assess the accuracy of varying risk scores composed of numerous clinical and/or angiographic variables to predict outcomes in patients with moderate- and high-risk NSTEACS undergoing PCI. Several traditional and novel measures were used to assess improvement in model performance. The main findings of this study are as follows: (1) scores incorporating both clinical and angiographic variables displayed the best tradeoff in terms of discrimination and calibration, showing the best c statistic and index of separation, a net reclassification improvement for all ischemic end points, with good calibration for most of them; (2) pure clinical scores had, in general, better calibration but less than satisfactory discrimination for most end points compared with the other scores; (3) the Clinical SYNTAX score emerged as the most accurate for risk prediction of 1-year cardiac mortality by all statistical tests; and (4) all scores had better prognostic accuracy for risk prediction of mortality or cardiac mortality than did the other ischemic outcomes.
Scoring systems currently used for risk stratification of patients with NSTEACS are based on multivariable models that integrate elements from the medical history, admission electrocardiogram, and biochemical evidence of myocyte necrosis. In addition, studies performed in stable or unstable patients with multivessel or left main disease have shown that the anatomical SYNTAX score independently predicts mortality and MACE. Retrospective studies have indicated that the ability of the SYNTAX score to predict outcomes may further be improved when combined with the Euroscore or the ACEF score. The considerable heterogeneity of these studies bears on the question of the relative accuracy of the varying risk scores for prediction of adverse outcomes in patients with NSTEACS undergoing PCI.
In the present study, the scores incorporating both clinical and angiographic variables had the best discrimination in relation to all end points considered, including 1-year death, cardiac death, myocardial infarction, target vessel revascularization, stent thrombosis, and MACE, and good calibration for most end points. Specifically, the NERS score provided the greatest discrimination for risk prediction of all-cause death, target vessel revascularization, and MACE. However, the NERS score requires evaluation of 54 variables (17 clinical, 4 procedural, and 33 angiographic), making the application of this score in a clinical setting difficult. A simpler risk score is required for widespread adoption. In this regard, the Clinical SYNTAX score is less complicated than the NERS score and had nearly equivalent discrimination with better calibration for all-cause death and even better performance for prediction of cardiac death and MI. In addition, the Clinical SYNTAX score had the best index of separation for most end points considered. Interestingly, all scores displayed better discrimination for risk prediction of mortality and cardiac mortality than other outcomes, for which, in general, the prognostic accuracy was less than satisfactory. This finding underscores the importance of using these scores for predicting individual rather than combined outcomes and that novel prognostication models should focus on specific outcomes, with mortality being the most important.
Calibration was, in general, better for the scores based purely on clinical variables, with the TIMI score showing the best results for most end points considered. However, a model with good calibration but poor discrimination is of limited clinical use. In this regard, the scores incorporating angiographic and clinical variables achieved a better tradeoff between discrimination and calibration than other scores in relation to most ischemic end points. Moreover, the scores incorporating angiographic and clinical variables displayed the best index of separation and net reclassification improvement for all end points.
Risk assessment of patients with NSTEACS should be a continuous process that integrates data acquired from the time of presentation through the index hospitalization. The TIMI and GRACE scores are valuable tools for initial stratification of patients with NSTEACS. However, these scores were not specifically optimized for patients with NSTEACS undergoing PCI, which may explain their poorer accuracy in terms of discrimination compared with scores incorporating angiographic variables. Of note, in our study, the GRACE score had a better prognostic accuracy in the whole cohort of patients than in PCI-treated patients. The present study demonstrates that angiographic factors add important prognostic value for the prediction of 1-year ischemic outcomes after PCI in NSTEACS. Not only were scores incorporating both clinical and angiographic variables stronger in terms of discrimination at predicting cardiac death and MI, but adding angiographic measures to clinical end points was also required for accurate prognostication of other important ischemic end points such as target vessel revascularization, stent thrombosis, and MACE. Therefore, scores that incorporate both clinical and angiographic variables have superior ability to comprehensively predict numerous adverse ischemic outcomes beyond just mortality.
Rather than originating from a derivation cohort with subsequent validation in a test cohort, the SYNTAX score was empirically developed based on an arbitrary ranking of lesion complexity. As such, further refinement of this purely anatomical score is theoretically possible by differently weighing the variables it incorporates or considering other angiographic variables such as TIMI flow. In this regard, the original SYNTAX score, as well as the Clinical SYNTAX and NERS scores, were the only ones to have good discrimination and good calibration for risk prognostication of cardiac death. However, the Clinical SYNTAX score had a better index of separation and net reclassification improvement than both the anatomical SYNTAX and the NERS scores. These findings clearly indicate that integrating angiographic and clinical variables is required to construct more accurate prognostic models for patients with NSTEACS undergoing PCI.
This report has several limitations that should be acknowledged. As a retrospective analysis from a prospective, randomized trial, the results should be considered hypothesis generating. All patients included in this study were at medium or high risk, and all underwent coronary angiography. Moreover, patients treated with CABG or medical therapy, as well as those with a history of prior CABG or major comorbidities, were excluded. However, our principal objective was to address the prognostic accuracy of several risk scores in the specific subset of patients with NSTEACS undergoing PCI; the present findings should not be generalized to other subgroups. In clinical practice, scores incorporating angiographic variables are calculated by visual lesion assessment (rather than by core laboratory determination as in the present report), which would likely lead to greater interobserver variability. Significant clinical and angiographic differences between patients included and excluded in this study were apparent. This was mainly due to the significant proportion of patients excluded for prior coronary artery bypass graft surgery, for whom the SYNTAX score could not be calculated. All total occlusions were scored as having unknown duration; many of these occlusions in ACUITY may have been of short duration, leading to overestimation of the SYNTAX score. Important differences between scores in prognostic accuracy were apparent, but the extent to which these differences impact clinical decision making and outcomes warrants further investigation.
In conclusion, in patients with NSTEACS undergoing PCI, risk scores incorporating clinical and angiographic variables had the highest discrimination for a broad spectrum of ischemic end points, including mortality, MI, target vessel revascularization, stent thrombosis, and MACE, and were reasonably well calibrated for most of these end points (although less so than scores based purely on clinical or anatomical variables alone). Depending on the end point considered, specific risk scores may be used to optimize predictive accuracy in clinical practice, although the Clinical SYNTAX score was the most accurate for risk prediction of 1-year cardiac mortality.
Discussion
Assessing the performance of prediction models is a complex process that requires the evaluation of several performance measures. Traditional measures for survival outcomes include discrimination, usually expressed as c statistic, and calibration, usually measured with the Hosmer-Lemeshow test. Discrimination is a measure of how well the prognostic model can separate cases from controls, whereas calibration is a measure of how well predicted probabilities fit actual risks. Typically, prognostic models cannot have both a perfect calibration and a perfect discrimination, and therefore, the best models will achieve an acceptable balance between these 2 measures. The c statistic, however, possesses inherent limitations because it is a function only of ranks and not of predicted probabilities, and therefore, it is insensitive to changes in absolute risk estimates. New measures have recently been proposed that offer incremental information over the c statistic. In this regard, the net reclassification improvement and the integrated discrimination index have recently been proposed as further measures to assess improvement in model performance. The present study is the first to assess the accuracy of varying risk scores composed of numerous clinical and/or angiographic variables to predict outcomes in patients with moderate- and high-risk NSTEACS undergoing PCI. Several traditional and novel measures were used to assess improvement in model performance. The main findings of this study are as follows: (1) scores incorporating both clinical and angiographic variables displayed the best tradeoff in terms of discrimination and calibration, showing the best c statistic and index of separation, a net reclassification improvement for all ischemic end points, with good calibration for most of them; (2) pure clinical scores had, in general, better calibration but less than satisfactory discrimination for most end points compared with the other scores; (3) the Clinical SYNTAX score emerged as the most accurate for risk prediction of 1-year cardiac mortality by all statistical tests; and (4) all scores had better prognostic accuracy for risk prediction of mortality or cardiac mortality than did the other ischemic outcomes.
Scoring systems currently used for risk stratification of patients with NSTEACS are based on multivariable models that integrate elements from the medical history, admission electrocardiogram, and biochemical evidence of myocyte necrosis. In addition, studies performed in stable or unstable patients with multivessel or left main disease have shown that the anatomical SYNTAX score independently predicts mortality and MACE. Retrospective studies have indicated that the ability of the SYNTAX score to predict outcomes may further be improved when combined with the Euroscore or the ACEF score. The considerable heterogeneity of these studies bears on the question of the relative accuracy of the varying risk scores for prediction of adverse outcomes in patients with NSTEACS undergoing PCI.
In the present study, the scores incorporating both clinical and angiographic variables had the best discrimination in relation to all end points considered, including 1-year death, cardiac death, myocardial infarction, target vessel revascularization, stent thrombosis, and MACE, and good calibration for most end points. Specifically, the NERS score provided the greatest discrimination for risk prediction of all-cause death, target vessel revascularization, and MACE. However, the NERS score requires evaluation of 54 variables (17 clinical, 4 procedural, and 33 angiographic), making the application of this score in a clinical setting difficult. A simpler risk score is required for widespread adoption. In this regard, the Clinical SYNTAX score is less complicated than the NERS score and had nearly equivalent discrimination with better calibration for all-cause death and even better performance for prediction of cardiac death and MI. In addition, the Clinical SYNTAX score had the best index of separation for most end points considered. Interestingly, all scores displayed better discrimination for risk prediction of mortality and cardiac mortality than other outcomes, for which, in general, the prognostic accuracy was less than satisfactory. This finding underscores the importance of using these scores for predicting individual rather than combined outcomes and that novel prognostication models should focus on specific outcomes, with mortality being the most important.
Calibration was, in general, better for the scores based purely on clinical variables, with the TIMI score showing the best results for most end points considered. However, a model with good calibration but poor discrimination is of limited clinical use. In this regard, the scores incorporating angiographic and clinical variables achieved a better tradeoff between discrimination and calibration than other scores in relation to most ischemic end points. Moreover, the scores incorporating angiographic and clinical variables displayed the best index of separation and net reclassification improvement for all end points.
Risk assessment of patients with NSTEACS should be a continuous process that integrates data acquired from the time of presentation through the index hospitalization. The TIMI and GRACE scores are valuable tools for initial stratification of patients with NSTEACS. However, these scores were not specifically optimized for patients with NSTEACS undergoing PCI, which may explain their poorer accuracy in terms of discrimination compared with scores incorporating angiographic variables. Of note, in our study, the GRACE score had a better prognostic accuracy in the whole cohort of patients than in PCI-treated patients. The present study demonstrates that angiographic factors add important prognostic value for the prediction of 1-year ischemic outcomes after PCI in NSTEACS. Not only were scores incorporating both clinical and angiographic variables stronger in terms of discrimination at predicting cardiac death and MI, but adding angiographic measures to clinical end points was also required for accurate prognostication of other important ischemic end points such as target vessel revascularization, stent thrombosis, and MACE. Therefore, scores that incorporate both clinical and angiographic variables have superior ability to comprehensively predict numerous adverse ischemic outcomes beyond just mortality.
Rather than originating from a derivation cohort with subsequent validation in a test cohort, the SYNTAX score was empirically developed based on an arbitrary ranking of lesion complexity. As such, further refinement of this purely anatomical score is theoretically possible by differently weighing the variables it incorporates or considering other angiographic variables such as TIMI flow. In this regard, the original SYNTAX score, as well as the Clinical SYNTAX and NERS scores, were the only ones to have good discrimination and good calibration for risk prognostication of cardiac death. However, the Clinical SYNTAX score had a better index of separation and net reclassification improvement than both the anatomical SYNTAX and the NERS scores. These findings clearly indicate that integrating angiographic and clinical variables is required to construct more accurate prognostic models for patients with NSTEACS undergoing PCI.
This report has several limitations that should be acknowledged. As a retrospective analysis from a prospective, randomized trial, the results should be considered hypothesis generating. All patients included in this study were at medium or high risk, and all underwent coronary angiography. Moreover, patients treated with CABG or medical therapy, as well as those with a history of prior CABG or major comorbidities, were excluded. However, our principal objective was to address the prognostic accuracy of several risk scores in the specific subset of patients with NSTEACS undergoing PCI; the present findings should not be generalized to other subgroups. In clinical practice, scores incorporating angiographic variables are calculated by visual lesion assessment (rather than by core laboratory determination as in the present report), which would likely lead to greater interobserver variability. Significant clinical and angiographic differences between patients included and excluded in this study were apparent. This was mainly due to the significant proportion of patients excluded for prior coronary artery bypass graft surgery, for whom the SYNTAX score could not be calculated. All total occlusions were scored as having unknown duration; many of these occlusions in ACUITY may have been of short duration, leading to overestimation of the SYNTAX score. Important differences between scores in prognostic accuracy were apparent, but the extent to which these differences impact clinical decision making and outcomes warrants further investigation.
In conclusion, in patients with NSTEACS undergoing PCI, risk scores incorporating clinical and angiographic variables had the highest discrimination for a broad spectrum of ischemic end points, including mortality, MI, target vessel revascularization, stent thrombosis, and MACE, and were reasonably well calibrated for most of these end points (although less so than scores based purely on clinical or anatomical variables alone). Depending on the end point considered, specific risk scores may be used to optimize predictive accuracy in clinical practice, although the Clinical SYNTAX score was the most accurate for risk prediction of 1-year cardiac mortality.
Source...