Journal Article
. 2020 Jan; 21(2):233-241.
doi: 10.1016/S1470-2045(19)30739-9.

Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study

Wouter Bulten 1 Hans Pinckaers 2 Hester van Boven 3 Robert Vink 4 Thomas de Bel 2 Bram van Ginneken 5 Jeroen van der Laak 2 Christina Hulsbergen-van de Kaa 4 Geert Litjens 2 
  • PMID: 31926805
  •     77 citations


Background: The Gleason score is the strongest correlating predictor of recurrence for prostate cancer, but has substantial inter-observer variability, limiting its usefulness for individual patients. Specialised urological pathologists have greater concordance; however, such expertise is not widely available. Prostate cancer diagnostics could thus benefit from robust, reproducible Gleason grading. We aimed to investigate the potential of deep learning to perform automated Gleason grading of prostate biopsies.

Methods: In this retrospective study, we developed a deep-learning system to grade prostate biopsies following the Gleason grading standard. The system was developed using randomly selected biopsies, sampled by the biopsy Gleason score, from patients at the Radboud University Medical Center (pathology report dated between Jan 1, 2012, and Dec 31, 2017). A semi-automatic labelling technique was used to circumvent the need for manual annotations by pathologists, using pathologists' reports as the reference standard during training. The system was developed to delineate individual glands, assign Gleason growth patterns, and determine the biopsy-level grade. For validation of the method, a consensus reference standard was set by three expert urological pathologists on an independent test set of 550 biopsies. Of these 550, 100 were used in an observer experiment, in which the system, 13 pathologists, and two pathologists in training were compared with respect to the reference standard. The system was also compared to an external test dataset of 886 cores, which contained 245 cores from a different centre that were independently graded by two pathologists.

Findings: We collected 5759 biopsies from 1243 patients. The developed system achieved a high agreement with the reference standard (quadratic Cohen's kappa 0·918, 95% CI 0·891-0·941) and scored highly at clinical decision thresholds: benign versus malignant (area under the curve 0·990, 95% CI 0·982-0·996), grade group of 2 or more (0·978, 0·966-0·988), and grade group of 3 or more (0·974, 0·962-0·984). In an observer experiment, the deep-learning system scored higher (kappa 0·854) than the panel (median kappa 0·819), outperforming 10 of 15 pathologist observers. On the external test dataset, the system obtained a high agreement with the reference standard set independently by two pathologists (quadratic Cohen's kappa 0·723 and 0·707) and within inter-observer variability (kappa 0·71).

Interpretation: Our automated deep-learning system achieved a performance similar to pathologists for Gleason grading and could potentially contribute to prostate cancer diagnosis. The system could potentially assist pathologists by screening biopsies, providing second opinions on grade group, and presenting quantitative measurements of volume percentages.

Funding: Dutch Cancer Society.

Tailored for Real-World: A Whole Slide Image Classification System Validated on Uncurated Multi-Site Data Emulating the Prospective Pathology Workload.
Julianna D Ianni, Rajath E Soans, +9 authors, Jason B Lee.
Sci Rep, 2020 Feb 23; 10(1). PMID: 32081956    Free PMC article.
Deep learning assessment of breast terminal duct lobular unit involution: Towards automated prediction of breast cancer risk.
Suzanne C Wetstein, Allison M Onken, +13 authors, Mitko Veta.
PLoS One, 2020 Apr 16; 15(4). PMID: 32294107    Free PMC article.
Histologic tissue components provide major cues for machine learning-based prostate cancer detection and grading on prostatectomy specimens.
Wenchao Han, Carol Johnson, +6 authors, Aaron D Ward.
Sci Rep, 2020 Jun 20; 10(1). PMID: 32555410    Free PMC article.
Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer.
Mark Kriegsmann, Christian Haag, +18 authors, Katharina Kriegsmann.
Cancers (Basel), 2020 Jun 21; 12(6). PMID: 32560475    Free PMC article.
Development and Validation of a Deep Learning Algorithm for Gleason Grading of Prostate Cancer From Biopsy Specimens.
Kunal Nagpal, Davis Foote, +23 authors, Craig H Mermel.
JAMA Oncol, 2020 Jul 24; 6(9). PMID: 32701148    Free PMC article.
A deep learning model to predict RNA-Seq expression of tumours from whole slide images.
Benoît Schmauch, Alberto Romagnoni, +11 authors, Gilles Wainrib.
Nat Commun, 2020 Aug 05; 11(1). PMID: 32747659    Free PMC article.
Radio-pathomic mapping model generated using annotations from five pathologists reliably distinguishes high-grade prostate cancer.
Sean D McGarry, John D Bukowy, +17 authors, Peter S LaViolette.
J Med Imaging (Bellingham), 2020 Sep 15; 7(5). PMID: 32923510    Free PMC article.
Automated detection of cribriform growth patterns in prostate histology images.
Pierre Ambrosini, Eva Hollemans, +3 authors, Frans Vos.
Sci Rep, 2020 Sep 12; 10(1). PMID: 32913202    Free PMC article.
Deep learning quantification of percent steatosis in donor liver biopsy frozen sections.
Lulu Sun, Jon N Marsh, +5 authors, Ta-Chiang Liu.
EBioMedicine, 2020 Sep 28; 60. PMID: 32980688    Free PMC article.
Identification of Gastritis Subtypes by Convolutional Neuronal Networks on Histological Images of Antrum and Corpus Biopsies.
Georg Steinbuss, Katharina Kriegsmann, Mark Kriegsmann.
Int J Mol Sci, 2020 Sep 17; 21(18). PMID: 32932860    Free PMC article.
Computer Extracted Features from Initial H&E Tissue Biopsies Predict Disease Progression for Prostate Cancer Patients on Active Surveillance.
Sacheth Chandramouli, Patrick Leo, +7 authors, Anant Madabhushi.
Cancers (Basel), 2020 Sep 25; 12(9). PMID: 32967377    Free PMC article.
Evaluation of the Use of Combined Artificial Intelligence and Pathologist Assessment to Review and Grade Prostate Biopsies.
David F Steiner, Kunal Nagpal, +18 authors, Craig H Mermel.
JAMA Netw Open, 2020 Nov 13; 3(11). PMID: 33180129    Free PMC article.
Digital Pathology: Advantages, Limitations and Emerging Perspectives.
Stephan W Jahn, Markus Plass, Farid Moinfar.
J Clin Med, 2020 Nov 22; 9(11). PMID: 33217963    Free PMC article.
Deep neural network models for computational histopathology: A survey.
Chetan L Srinidhi, Ozan Ciga, Anne L Martel.
Med Image Anal, 2020 Oct 14; 67. PMID: 33049577    Free PMC article.
Reimagining T Staging Through Artificial Intelligence and Machine Learning Image Processing Approaches in Digital Pathology.
Kaustav Bera, Ian Katz, Anant Madabhushi.
JCO Clin Cancer Inform, 2020 Nov 10; 4. PMID: 33166198    Free PMC article.
Artificial intelligence (AI) in urology-Current use and future directions: An iTRUE study.
Milap Shah, Nithesh Naik, Bhaskar K Somani, B M Zeeshan Hameed.
Turk J Urol, 2020 Jun 02; 46(Supp. 1). PMID: 32479253    Free PMC article.
Single-Cell Spatial Analysis of Tumor and Immune Microenvironment on Whole-Slide Image Reveals Hepatocellular Carcinoma Subtypes.
Haiyue Wang, Yuming Jiang, +3 authors, Ruijiang Li.
Cancers (Basel), 2020 Dec 03; 12(12). PMID: 33260561    Free PMC article.
A 'Real-Life' Experience on Automated Digital Image Analysis of FGFR2 Immunohistochemistry in Breast Cancer.
Marcin Braun, Dominika Piasecka, +3 authors, Hanna M Romanska.
Diagnostics (Basel), 2020 Dec 11; 10(12). PMID: 33297384    Free PMC article.
Explainable Deep Learning for Pulmonary Disease and Coronavirus COVID-19 Detection from X-rays.
Luca Brunese, Francesco Mercaldo, Alfonso Reginelli, Antonella Santone.
Comput Methods Programs Biomed, 2020 Jul 01; 196. PMID: 32599338    Free PMC article.
Designing deep learning studies in cancer diagnostics.
Andreas Kleppe, Ole-Johan Skrede, +3 authors, Håvard E Danielsen.
Nat Rev Cancer, 2021 Jan 31; 21(3). PMID: 33514930
Aggregation of cohorts for histopathological diagnosis with deep morphological analysis.
Jeonghyuk Park, Yul Ri Chung, +5 authors, Kyu-Hwan Jung.
Sci Rep, 2021 Feb 05; 11(1). PMID: 33536550    Free PMC article.
Agreement of two pre-trained deep-learning neural networks built with transfer learning with six pathologists on 6000 patches of prostate cancer from Gleason2019 Challenge.
Mircea Sebastian Şerbănescu, Carmen Nicoleta Oancea, +4 authors, Răzvan Mihail Pleşea.
Rom J Morphol Embryol, 2021 Feb 06; 61(2). PMID: 33544803    Free PMC article.
Deep learning in cancer pathology: a new generation of clinical biomarkers.
Amelie Echle, Niklas Timon Rindtorff, +3 authors, Jakob Nikolas Kather.
Br J Cancer, 2020 Nov 19; 124(4). PMID: 33204028    Free PMC article.
Deep learning-based grading of ductal carcinoma in situ in breast histopathology images.
Suzanne C Wetstein, Nikolas Stathonikos, +5 authors, Mitko Veta.
Lab Invest, 2021 Feb 21; 101(4). PMID: 33608619    Free PMC article.
Contemporary application of artificial intelligence in prostate cancer: an i-TRUE study.
B M Zeeshan Hameed, Milap Shah, +5 authors, Bhavan Prasad Rai.
Ther Adv Urol, 2021 Feb 27; 13. PMID: 33633799    Free PMC article.
Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists.
Wouter Bulten, Maschenka Balkenhol, +20 authors, Geert Litjens.
Mod Pathol, 2020 Aug 08; 34(3). PMID: 32759979    Free PMC article.
CovidCTNet: an open-source deep learning approach to diagnose covid-19 using small cohort of CT images.
Tahereh Javaheri, Morteza Homayounfar, +19 authors, Reza Rawassizadeh.
NPJ Digit Med, 2021 Feb 20; 4(1). PMID: 33603193    Free PMC article.
Artificial Intelligence and Machine Learning in Prostate Cancer Patient Management-Current Trends and Future Perspectives.
Octavian Sabin Tătaru, Mihai Dorin Vartolomei, +12 authors, Matteo Ferro.
Diagnostics (Basel), 2021 Mar 07; 11(2). PMID: 33672608    Free PMC article.
Epidemiology and genomics of prostate cancer in Asian men.
Yao Zhu, Miao Mo, +5 authors, Dingwei Ye.
Nat Rev Urol, 2021 Mar 12; 18(5). PMID: 33692499
A multi-resolution model for histopathology image classification and localization with multiple instance learning.
Jiayun Li, Wenyuan Li, +4 authors, Corey W Arnold.
Comput Biol Med, 2021 Feb 19; 131. PMID: 33601084    Free PMC article.
Accurate diagnosis of colorectal cancer based on histopathology images using artificial intelligence.
K S Wang, G Yu, +51 authors, H W Deng.
BMC Med, 2021 Mar 24; 19(1). PMID: 33752648    Free PMC article.
Current and future perspectives of digital microscopy with fluorescence confocal microscope for prostate tissue interpretation: a narrative review.
Bernardo Rocco, Alessia Cimadamore, +13 authors, Maria Chiara Sighinolfi.
Transl Androl Urol, 2021 Apr 15; 10(3). PMID: 33850791    Free PMC article.
Computerized tumor multinucleation index (MuNI) is prognostic in p16+ oropharyngeal carcinoma.
Can F Koyuncu, Cheng Lu, +21 authors, Anant Madabhushi.
J Clin Invest, 2021 Mar 03; 131(8). PMID: 33651718    Free PMC article.
Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology.
Cesare Lancellotti, Pierandrea Cancian, +4 authors, Luca Di Tommaso.
Cells, 2021 May 01; 10(4). PMID: 33918173    Free PMC article.
Computer extracted gland features from H&E predicts prostate cancer recurrence comparably to a genomic companion diagnostic test: a large multi-site study.
Patrick Leo, Andrew Janowczyk, +29 authors, Anant Madabhushi.
NPJ Precis Oncol, 2021 May 05; 5(1). PMID: 33941830    Free PMC article.
Combining weakly and strongly supervised learning improves strong supervision in Gleason pattern classification.
Sebastian Otálora, Niccolò Marini, Henning Müller, Manfredo Atzori.
BMC Med Imaging, 2021 May 10; 21(1). PMID: 33964886    Free PMC article.
Deep learning in histopathology: the path to the clinic.
Jeroen van der Laak, Geert Litjens, Francesco Ciompi.
Nat Med, 2021 May 16; 27(5). PMID: 33990804
Europe Unites for the Digital Transformation of Pathology: The Role of the New ESDIP.
Catarina Eloy, Norman Zerbe, Filippo Fraggetta.
J Pathol Inform, 2021 May 21; 12. PMID: 34012714    Free PMC article.
Automated gleason grading on prostate biopsy slides by statistical representations of homology profile.
Chaoyang Yan, Kazuaki Nakane, +6 authors, Jun Xu.
Comput Methods Programs Biomed, 2020 May 30; 194. PMID: 32470903    Free PMC article.
Deep Learning in Pancreatic Tissue: Identification of Anatomical Structures, Pancreatic Intraepithelial Neoplasia, and Ductal Adenocarcinoma.
Mark Kriegsmann, Katharina Kriegsmann, +3 authors, Matthias M Gaida.
Int J Mol Sci, 2021 Jun 03; 22(10). PMID: 34065423    Free PMC article.
Federated learning improves site performance in multicenter deep learning without data sharing.
Karthik V Sarma, Stephanie Harmon, +16 authors, Corey W Arnold.
J Am Med Inform Assoc, 2021 Feb 05; 28(6). PMID: 33537772    Free PMC article.
Yet Another Automated Gleason Grading System (YAAGGS) by weakly supervised deep learning.
Yechan Mun, Inyoung Paik, +2 authors, Hyeyoon Chang.
NPJ Digit Med, 2021 Jun 16; 4(1). PMID: 34127777    Free PMC article.
An automated computational image analysis pipeline for histological grading of cardiac allograft rejection.
Eliot G Peyster, Sara Arabyarmohammadi, +9 authors, Anant Madabhushi.
Eur Heart J, 2021 May 14; 42(24). PMID: 33982079    Free PMC article.
Independent real-world application of a clinical-grade automated prostate cancer detection system.
Leonard M da Silva, Emilio M Pereira, +18 authors, Jorge S Reis-Filho.
J Pathol, 2021 Apr 28; 254(2). PMID: 33904171    Free PMC article.
The impact of site-specific digital histology signatures on deep learning model accuracy and bias.
Frederick M Howard, James Dolezal, +10 authors, Alexander T Pearson.
Nat Commun, 2021 Jul 22; 12(1). PMID: 34285218    Free PMC article.
Development of a Deep-Learning Pipeline to Recognize and Characterize Macrophages in Colo-Rectal Liver Metastasis.
Pierandrea Cancian, Nina Cortese, +12 authors, Luca Di Tommaso.
Cancers (Basel), 2021 Jul 21; 13(13). PMID: 34282750    Free PMC article.
Alterations in protein expression and site-specific N-glycosylation of prostate cancer tissues.
Simon Sugár, Gábor Tóth, +4 authors, Lilla Turiák.
Sci Rep, 2021 Aug 07; 11(1). PMID: 34354152    Free PMC article.
Artificial intelligence for advance requesting of immunohistochemistry in diagnostically uncertain prostate biopsies.
Andrea Chatrian, Richard T Colling, +11 authors, Clare Verrill.
Mod Pathol, 2021 May 22; 34(9). PMID: 34017063    Free PMC article.
Computationally Derived Cribriform Area Index from Prostate Cancer Hematoxylin and Eosin Images Is Associated with Biochemical Recurrence Following Radical Prostatectomy and Is Most Prognostic in Gleason Grade Group 2.
Patrick Leo, Sacheth Chandramouli, +19 authors, Anant Madabhushi.
Eur Urol Focus, 2021 May 05; 7(4). PMID: 33941504    Free PMC article.
Improving Generalization of Deep Learning Models for Diagnostic Pathology by Increasing Variability in Training Data: Experiments on Osteosarcoma Subtypes.
Haiming Tang, Nanfei Sun, Steven Shen.
J Pathol Inform, 2021 Sep 10; 12. PMID: 34497734    Free PMC article.
PTEN and DNA Ploidy Status by Machine Learning in Prostate Cancer.
Karolina Cyll, Andreas Kleppe, +11 authors, Håvard E Danielsen.
Cancers (Basel), 2021 Sep 11; 13(17). PMID: 34503100    Free PMC article.
COVID-19 challenges and its therapeutics.
Sabi Ur Rehman, Shaheed Ur Rehman, Hye Hyun Yoo.
Biomed Pharmacother, 2021 Aug 14; 142. PMID: 34388532    Free PMC article.
Identification of nodal micrometastasis in colorectal cancer using deep learning on annotation-free whole-slide images.
Wen-Yu Chuang, Chi-Chung Chen, +7 authors, Chao-Yuan Yeh.
Mod Pathol, 2021 Jun 10; 34(10). PMID: 34103664    Free PMC article.
MRI and Targeted Biopsy Essential Tools for an Accurate Diagnosis and Treatment Decision Making in Prostate Cancer.
Suraj Samtani, Mauricio Burotto, +2 authors, Annerleim Walton-Diaz.
Diagnostics (Basel), 2021 Sep 29; 11(9). PMID: 34573893    Free PMC article.
Robust whole slide image analysis for cervical cancer screening using deep learning.
Shenghua Cheng, Sibo Liu, +20 authors, Xiuli Liu.
Nat Commun, 2021 Sep 26; 12(1). PMID: 34561435    Free PMC article.
Applied machine learning in cancer research: A systematic review for patient diagnosis, classification and prognosis.
Konstantina Kourou, Konstantinos P Exarchos, +3 authors, Dimitrios I Fotiadis.
Comput Struct Biotechnol J, 2021 Oct 30; 19. PMID: 34712399    Free PMC article.
Development and Validation of an Artificial Intelligence-Powered Platform for Prostate Cancer Grading and Quantification.
Wei Huang, Ramandeep Randhawa, +6 authors, Rajat Roy.
JAMA Netw Open, 2021 Nov 04; 4(11). PMID: 34730818    Free PMC article.
Significant Inter- and Intralaboratory Variation in Gleason Grading of Prostate Cancer: A Nationwide Study of 35,258 Patients in The Netherlands.
Rachel N Flach, Peter-Paul M Willemse, +5 authors, Richard P Meijer.
Cancers (Basel), 2021 Nov 14; 13(21). PMID: 34771542    Free PMC article.
Quality control stress test for deep learning-based diagnostic model in digital pathology.
Birgid Schömig-Markiefka, Alexey Pryalukhin, +8 authors, Yuri Tolkach.
Mod Pathol, 2021 Jun 26; 34(12). PMID: 34168282    Free PMC article.
Deep Learning-Based Artificial Intelligence System for Automatic Assessment of Glomerular Pathological Findings in Lupus Nephritis.
Zhaohui Zheng, Xiangsen Zhang, +5 authors, Ping Zhu.
Diagnostics (Basel), 2021 Nov 28; 11(11). PMID: 34829330    Free PMC article.
Recommendations for diagnosing STIC: a systematic review and meta-analysis.
Joep M A Bogaerts, Miranda P Steenbeek, +4 authors, Michiel Simons.
Virchows Arch, 2021 Dec 02; 480(4). PMID: 34850262    Free PMC article.
Systematic Review.
Selective identification and localization of indolent and aggressive prostate cancers via CorrSigNIA: an MRI-pathology correlation and deep learning framework.
Indrani Bhattacharya, Arun Seetharaman, +10 authors, Mirabela Rusu.
Med Image Anal, 2021 Nov 17; 75. PMID: 34784540    Free PMC article.
Data-efficient and weakly supervised computational pathology on whole-slide images.
Ming Y Lu, Drew F K Williamson, +3 authors, Faisal Mahmood.
Nat Biomed Eng, 2021 Mar 03; 5(6). PMID: 33649564    Free PMC article.
Non-invasive scoring of cellular atypia in keratinocyte cancers in 3D LC-OCT images using Deep Learning.
Sébastien Fischman, Javiera Pérez-Anker, +11 authors, Jean-Luc Perrot.
Sci Rep, 2022 Jan 12; 12(1). PMID: 35013485    Free PMC article.
A data-driven ultrasound approach discriminates pathological high grade prostate cancer.
Jun Akatsuka, Yasushi Numata, +11 authors, Yoichiro Yamamoto.
Sci Rep, 2022 Jan 19; 12(1). PMID: 35039648    Free PMC article.
Automated quantification of levels of breast terminal duct lobular (TDLU) involution using deep learning.
Thomas de Bel, Geert Litjens, +14 authors, Jeroen A W M van der Laak.
NPJ Breast Cancer, 2022 Jan 21; 8(1). PMID: 35046392    Free PMC article.
Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge.
Wouter Bulten, Kimmo Kartasalo, +31 authors, PANDA challenge consortium.
Nat Med, 2022 Jan 15; 28(1). PMID: 35027755    Free PMC article.
Study protocol: retrospectively mining multisite clinical data to presymptomatically predict seizure onset for individual patients with Sturge-Weber.
Pooja Vedmurthy, Anna L R Pinto, +3 authors, BCH-KKI SWS Pre-symptomatic Biomarker Working Group.
BMJ Open, 2022 Feb 06; 12(2). PMID: 35121603    Free PMC article.
Code-Free Development and Deployment of Deep Segmentation Models for Digital Pathology.
Henrik Sahlin Pettersen, Ilya Belevich, +6 authors, André Pedersen.
Front Med (Lausanne), 2022 Feb 15; 8. PMID: 35155486    Free PMC article.
Computer-Aided Assessment of Melanocytic Lesions by Means of a Mitosis Algorithm.
Bart Sturm, David Creytens, +7 authors, Jeroen A W M van der Laak.
Diagnostics (Basel), 2022 Feb 26; 12(2). PMID: 35204526    Free PMC article.
Automated Diagnosis of Cervical Intraepithelial Neoplasia in Histology Images via Deep Learning.
Bum-Joo Cho, Jeong-Won Kim, +6 authors, Sung-Taek Park.
Diagnostics (Basel), 2022 Feb 26; 12(2). PMID: 35204638    Free PMC article.
A deep learning system for prostate cancer diagnosis and grading in whole slide images of core needle biopsies.
Nitin Singhal, Shailesh Soni, +8 authors, Arvind Ganpule.
Sci Rep, 2022 Mar 03; 12(1). PMID: 35233002    Free PMC article.
A statistical, voxelised model of prostate cancer for biologically optimised radiotherapy.
Robert N Finnegan, Hayley M Reynolds, +8 authors, Annette Haworth.
Phys Imaging Radiat Oncol, 2022 Mar 15; 21. PMID: 35284663    Free PMC article.
The Global Research of Artificial Intelligence on Prostate Cancer: A 22-Year Bibliometric Analysis.
Zefeng Shen, Haiyang Wu, +4 authors, Tianxin Lin.
Front Oncol, 2022 Mar 19; 12. PMID: 35299747    Free PMC article.
A semi-supervised learning framework for micropapillary adenocarcinoma detection.
Yuan Gao, Yanhui Ding, +5 authors, Yuanjie Zheng.
Int J Comput Assist Radiol Surg, 2022 Feb 13; 17(4). PMID: 35149953
Fine-Tuned DenseNet-169 for Breast Cancer Metastasis Prediction Using FastAI and 1-Cycle Policy.
Adarsh Vulli, Parvathaneni Naga Srinivasu, +3 authors, Muhammad Fazal Ijaz.
Sensors (Basel), 2022 Apr 24; 22(8). PMID: 35458972    Free PMC article.
Computer-extracted features of nuclear morphology in hematoxylin and eosin images distinguish stage II and IV colon tumors.
Neeraj Kumar, Ruchika Verma, +4 authors, Anant Madabhushi.
J Pathol, 2022 Jan 11; 257(1). PMID: 35007352    Free PMC article.