Accuracy of the Vancouver Lung Cancer Risk Prediction Model Compared With That of Radiologists

      Background

      Risk models have been developed that include the subject’s pretest risk profile and imaging findings to predict the risk of cancer in an objective way. We assessed the accuracy of the Vancouver Lung Cancer Risk Prediction Model compared with that of trainee and experienced radiologists using a subset of size-matched nodules from the National Lung Screening Trial (NLST).

      Methods

      One hundred cases from the NLST database were selected (size range, 4-20 mm), including 20 proven cancers and 80 size-matched benign nodules. Three experienced thoracic radiologists and three trainee radiologists were asked to estimate the likelihood of cancer in each case, first independently, and then with knowledge of the model’s risk prediction. The results generated by the model alone also were estimated using receiver operating characteristic (ROC) analysis. The area under the ROC curve (AUC) for each viewing condition was calculated, and statistical significance in their differences was tested by using the Dorfman-Berbaum-Metz method.

      Results

      Human observers were more accurate (AUC value of 0.85 ± 0.05 [SD]) than was the model (0.77 ± 0.06) in estimating the risk of malignancy ( P = .0010), and use of the model did not improve their accuracy (0.84 ± 0.06). Experienced radiologists performed better than did trainees. Human observers could distinguish benign from malignant nodule morphology more accurately than could the model, which relies mainly on nodule size for risk estimation.

      Conclusions

      Experienced and trainee radiologists had superior ability to predict the risk of cancer in size-matched nodules from a screening trial compared with that of the Vancouver model, and use of the model did not improve their accuracy.

      Key Words

      Abbreviations:

      AUC ( area under the ROC curve), BCCA ( British Columbia Cancer Agency), NLST ( National Lung Screening Trial), PanCan ( Pan-Canadian Early Detection of Lung Cancer Study), ROC ( receiver operating characteristic)
      To read this article in full you will need to make a payment
      Subscribe to CHEST
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Aberle D.R.
        • Adams A.M.
        • et al.
        • National Lung Screening Trial Research Team
        Reduced lung-cancer mortality with low-dose computed tomographic screening.
        N Engl J Med. 2011; 365: 395-409
        • Church T.R.
        • Black W.C.
        • National Lung Screening Trial Research Team
        Results of initial low-dose computed tomographic screening for lung cancer.
        N Engl J Med. 2013; 368: 1980-1991
        • Aberle D.R.
        • DeMello S.
        • Berg C.D.
        • et al.
        • National Lung Screening Trial Research Team
        Results of the two incidence screenings in the National Lung Screening Trial.
        N Engl J Med. 2013; 369: 920-931
        • Detterbeck F.C.
        • Mazzone P.J.
        • Naidich D.P.
        • Back P.B.
        Screening for lung cancer: diagnosis and management of lung cancer—American College of Chest Physicians evidence-based clinical practice guidelines.
        Chest. 2013; 143: e78S-e92S
        • Horeweg N.
        • van Rosmalen J.
        • Heuvelmans M.A.
        • et al.
        Lung cancer probability in patients with CT-detected pulmonary nodules: a prespecified analysis of data from the NELSON trial of low-dose CT screening.
        Lancet Oncol. 2014; 15: 1332-1341
        • McWilliams A.
        • Tammemagi M.C.
        • Mayo J.R.
        • et al.
        Probability of cancer in pulmonary nodules detected on first screening CT.
        N Engl J Med. 2013; 369: 910-919
        • Pinsky P.F.
        • Gierada D.S.
        • Black W.
        • et al.
        Performance of lung-RADS in the National Lung Screening Trial.
        Ann Intern Med. 2015; 162: 485-491
        • American College of Radiology
        Lung CT screening reporting & data system.
        • Yip R.
        • Henschke C.I.
        • Yankelevitz D.F.
        • et al.
        CT screening for lung cancer: alternative definitions of positive test result based on the National Lung Screening Trial and International Early Lung Cancer Action Program databases.
        Radiology. 2014; 273: 591-596
        • Perandini S.
        • Soardi G.A.
        • Motton M.
        • Rossi A.
        • Signorini M.
        • Montemezzi S.
        Solid pulmonary nodule risk assessment and decision analysis: comparison of four prediction models in 285 cases.
        Eur Radiol. 2016; 26: 3071-3076
        • Soardi G.A.
        • Perandini S.
        • Motton M.
        • Montemezzi S.
        Assessing probability of malignancy in solid solitary pulmonary nodules with a new Bayesian calculator: improving diagnostic accuracy by means of expanded and updated features.
        Eur Radiol. 2015; 25: 155-162
        • Cassidy A.
        • Myles J.P.
        • van Tongeren M.
        • et al.
        The LLP risk model: an individual risk prediction model for lung cancer.
        Br J Cancer. 2008; 98: 270-276
        • White C.S.
        • Dharaiya E.
        • Campbell E.
        • Boroczky
        The Vancouver Lung Cancer Risk Prediction Model: assessment by using a subset of the National Lung Screening Trial cohort.
        Radiology. 2017; 283: 264-272
        • Wille M.M.W.
        • van Riel S.J.
        • Saghir Z.
        • et al.
        Predictive accuracy of the PanCan lung cancer risk prediction model: external validation based on CT from the Danish Lung Cancer Screening Trial.
        Eur Radiol. 2015; 25: 3093-3099
        • Hillis S.L.
        • Berbaun K.S.
        • Metz C.E.
        Recent development in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis.
        Acad Radiol. 2008; 15: 647-661
        • Metz C.E.
        • Pan X.
        “Proper” binormal ROC curves: theory and maximum-likelihood estimation.
        J Math Psychol. 1999; 43: 1-33
        • van Riel S.J.
        • Ciompi F.
        • Winkler Wille M.M.
        • et al.
        Malignancy risk estimation of pulmonary nodules in screening CTs: comparison between a computer model and human observers.
        PLoS One. 2017; 12e0185032
        • Kobayashi T.
        • Xu X.W.
        • MacMahon H.
        • Metz C.E.
        • Doi K.
        Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs.
        Radiology. 1996; 199: 843-848
        • Li F.
        • Aoyama M.
        • Shiraishi J.
        • et al.
        Radiologists’ performance for differentiating benign from malignant nodules on high-resolution CT using computer-estimated likelihood of malignancy.
        AJR Am J Roentgenol. 2004; 183: 1209-1215
        • Erasmus J.J.
        • Connolly J.E.
        • McAdams H.P.
        • Roggli V.L.
        Solitary pulmonary nodules: part 1—morphologic evaluation for differentiation of benign and malignant lesions.
        RadioGraphics. 2000; 20: 43-58
        • Truong M.T.
        • Sabloff B.S.
        • Ko J.P.
        Multidetector CT of solitary pulmonary nodules.
        Radiol Clin North Am. 2010; 48: 141-155
        • Chung K.
        • Jacobs C.
        • Scholten E.T.
        • et al.
        Lung-RADS category 4X: does it improve prediction of malignancy in subsolid nodules?.
        Radiology. 2017; 284: 264-271

      Linked Article

      • Response
        CHESTVol. 156Issue 4
        • In Brief
          We thank Drs Lam, Mayo, and Tammemagi for their comments on our article.1 We do not disagree with their assertion that the Vancouver model may play a useful role in determining the need for CT follow-up imaging of small nodules in the context of a screening program, or that future incorporation of radiomic features and artificial intelligence could improve the model’s accuracy. As noted in our article, it was necessary to use an enriched population sample with a relatively high proportion of cancers to perform a meaningful observer test, and we adjusted the model’s output to take this approach into account.
        • Full-Text
        • PDF
      • Human Observer vs Prediction Model for Lung Nodule Malignancy Risk Estimation
        CHESTVol. 156Issue 4
        • In Brief
          We read with interest the paper by MacMahon et al1 in an issue of CHEST (July 2019) comparing the accuracy of experienced thoracic radiologists and trainees vs the PanCan (Vancouver) Risk Prediction model2 to estimate the likelihood of malignancy of lung nodules found by using screening CT imaging.
        • Full-Text
        • PDF