Journal Article
. 2018 Sep; 4:30.
doi: 10.1038/s41523-018-0079-1.

Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype

Heather D Couture 1 Lindsay A Williams 2 Joseph Geradts 3 Sarah J Nyante 4 Ebonee N Butler 2 J S Marron 5 Charles M Perou 5 Melissa A Troester 2 Marc Niethammer 1 
  • PMID: 30182055
  •     30 References
  •     24 citations


RNA-based, multi-gene molecular assays are available and widely used for patients with ER-positive/HER2-negative breast cancers. However, RNA-based genomic tests can be costly and are not available in many countries. Methods for inferring molecular subtype from histologic images may identify patients most likely to benefit from further genomic testing. To identify patients who could benefit from molecular testing based on H&E stained histologic images, we developed an image analysis approach using deep learning. A training set of 571 breast tumors was used to create image-based classifiers for tumor grade, ER status, PAM50 intrinsic subtype, histologic subtype, and risk of recurrence score (ROR-PT). The resulting classifiers were applied to an independent test set (n = 288), and accuracy, sensitivity, and specificity of each was assessed on the test set. Histologic image analysis with deep learning distinguished low-intermediate vs. high tumor grade (82% accuracy), ER status (84% accuracy), Basal-like vs. non-Basal-like (77% accuracy), Ductal vs. Lobular (94% accuracy), and high vs. low-medium ROR-PT score (75% accuracy). Sampling considerations in the training set minimized bias in the test set. Incorrect classification of ER status was significantly more common for Luminal B tumors. These data provide proof of principle that molecular marker status, including a critical clinical biomarker (i.e., ER status), can be predicted with accuracy >75% based on H&E features. Image-based methods could be promising for identifying patients with a greater need for further genomic testing, or in place of classically scored variables typically accomplished using human-based scoring.

Systematic analysis of breast cancer morphology uncovers stromal features associated with survival.
Andrew H Beck, Ankur R Sangoi, +6 authors, Daphne Koller.
Sci Transl Med, 2011 Nov 11; 3(108). PMID: 22072638
Highly Cited.
Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling.
Yinyin Yuan, Henrik Failmezger, +14 authors, Florian Markowetz.
Sci Transl Med, 2012 Oct 27; 4(157). PMID: 23100629
Highly Cited.
Diversity of Breast Carcinoma: Histological Subtypes and Clinical Relevance.
Jaafar Makki.
Clin Med Insights Pathol, 2016 Jan 08; 8. PMID: 26740749    Free PMC article.
Highly Cited. Review.
Hormone receptor status, tumor characteristics, and prognosis: a prospective cohort of breast cancer patients.
Lisa K Dunnwald, Mary Anne Rossing, Christopher I Li.
Breast Cancer Res, 2007 Jan 24; 9(1). PMID: 17239243    Free PMC article.
Highly Cited.
Practical implications of gene-expression-based assays for breast oncologists.
Aleix Prat, Matthew J Ellis, Charles M Perou.
Nat Rev Clin Oncol, 2011 Dec 07; 9(1). PMID: 22143140    Free PMC article.
Highly Cited. Review.
Morphometic analysis of TCGA glioblastoma multiforme.
Hang Chang, Gerald V Fontenay, +5 authors, Bahram Parvin.
BMC Bioinformatics, 2011 Dec 22; 12. PMID: 22185703    Free PMC article.
A Global Covariance Descriptor for Nuclear Atypia Scoring in Breast Histopathology Images.
Adnan Mujahid Khan, Korsuk Sirinukunwattana, Nasir Rajpoot.
IEEE J Biomed Health Inform, 2015 Jun 23; 19(5). PMID: 26099150
A Deep Convolutional Neural Network for segmenting and classifying epithelial and stromal regions in histopathological images.
Jun Xu, Xiaofei Luo, +2 authors, Anant Madabhushi.
Neurocomputing, 2017 Feb 06; 191. PMID: 28154470    Free PMC article.
Highly Cited.
Assessment of algorithms for mitosis detection in breast cancer histopathology images.
Mitko Veta, Paul J van Diest, +26 authors, Josien P W Pluim.
Med Image Anal, 2014 Dec 31; 20(1). PMID: 25547073
Supervised risk predictor of breast cancer based on intrinsic subtypes.
Joel S Parker, Michael Mullins, +17 authors, Philip S Bernard.
J Clin Oncol, 2009 Feb 11; 27(8). PMID: 19204204    Free PMC article.
Highly Cited.
[Importance of a second opinion in breast surgical pathology and therapeutic implications].
Márcio de Almeida Salles, Fernanda Squarcio Fernandes Sanches, Amanda Arantes Perez, Helenice Gobbi.
Rev Bras Ginecol Obstet, 2009 Feb 17; 30(12). PMID: 19219341
Racial Differences in PAM50 Subtypes in the Carolina Breast Cancer Study.
Melissa A Troester, Xuezheng Sun, +16 authors, Charles M Perou.
J Natl Cancer Inst, 2017 Sep 02; 110(2). PMID: 28859290    Free PMC article.
Interobserver agreement and reproducibility in classification of invasive breast carcinoma: an NCI breast cancer family registry study.
Teri A Longacre, Marguerite Ennis, +10 authors, Frances P O'malley.
Mod Pathol, 2005 Dec 13; 19(2). PMID: 16341153
A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection.
Angel Alfonso Cruz-Roa, John Edison Arevalo Ovalle, Anant Madabhushi, Fabio Augusto González Osorio.
Med Image Comput Comput Assist Interv, 2014 Mar 01; 16(Pt 2). PMID: 24579166
Appearance Normalization of Histology Slides.
Marc Niethammer, David Borland, +2 authors, Nancy E Thomas.
Mach Learn Med Imaging, 2010 Jan 01; 6357. PMID: 25360444    Free PMC article.
Development of the 21-gene assay and its application in clinical practice and clinical trials.
Joseph A Sparano, Soonmyung Paik.
J Clin Oncol, 2008 Feb 09; 26(5). PMID: 18258979
Highly Cited. Review.
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Nima Tajbakhsh, Jae Y Shin, +4 authors, Jianming Liang.
IEEE Trans Med Imaging, 2016 Mar 16; 35(5). PMID: 26978662
Highly Cited.
Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases.
Andrew Janowczyk, Anant Madabhushi.
J Pathol Inform, 2016 Aug 27; 7. PMID: 27563488    Free PMC article.
Highly Cited.
Classification of Histology Sections via Multispectral Convolutional Sparse Coding.
Yin Zhou, Hang Chang, +2 authors, Bahram Parvin.
Conf Comput Vis Pattern Recognit Workshops, 2015 Jan 03; 2014. PMID: 25554749    Free PMC article.
Histopathological Image Classification Using Discriminative Feature-Oriented Dictionary Learning.
Tiep Huu Vu, Hojjat Seyed Mousavi, +2 authors, U K Arvind Rao.
IEEE Trans Med Imaging, 2015 Oct 30; 35(3). PMID: 26513781    Free PMC article.
Joint analysis of histopathology image features and gene expression in breast cancer.
Vlad Popovici, Eva Budinská, +4 authors, Rolf Jaggi.
BMC Bioinformatics, 2016 May 14; 17(1). PMID: 27170365    Free PMC article.
The impact of the Oncotype Dx breast cancer assay in clinical practice: a systematic review and meta-analysis.
Josh J Carlson, Joshua A Roth.
Breast Cancer Res Treat, 2013 Aug 27; 141(1). PMID: 23974828    Free PMC article.
Highly Cited. Systematic Review.
Histologic grading in breast cancer--reproducibility between seven pathologic departments. South Sweden Breast Cancer Group.
P Boiesen, P O Bendahl, +8 authors, M Fernö.
Acta Oncol, 2001 Feb 07; 39(1). PMID: 10752652
Image and statistical analysis of melanocytic histology.
Jayson Miedema, James Stephen Marron, +6 authors, Nancy E Thomas.
Histopathology, 2012 Jun 13; 61(3). PMID: 22687043    Free PMC article.
Performance of Three-Biomarker Immunohistochemistry for Intrinsic Breast Cancer Subtyping in the AMBER Consortium.
Emma H Allott, Stephanie M Cohen, +19 authors, Melissa A Troester.
Cancer Epidemiol Biomarkers Prev, 2015 Dec 30; 25(3). PMID: 26711328    Free PMC article.
Breast cancer receptor status: do results from a centralized pathology laboratory agree with SEER registry reports?
Huiyan Ma, Yaping Wang, +11 authors, Leslie Bernstein.
Cancer Epidemiol Biomarkers Prev, 2009 Aug 08; 18(8). PMID: 19661080    Free PMC article.
Integrated morphologic analysis for the identification and characterization of disease subtypes.
Lee A D Cooper, Jun Kong, +12 authors, Joel H Saltz.
J Am Med Inform Assoc, 2012 Jan 27; 19(2). PMID: 22278382    Free PMC article.
Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification.
Le Hou, Dimitris Samaras, +3 authors, Joel H Saltz.
Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, 2016 Nov 01; 2016. PMID: 27795661    Free PMC article.
Highly Cited.
Race, breast cancer subtypes, and survival in the Carolina Breast Cancer Study.
Lisa A Carey, Charles M Perou, +14 authors, Robert C Millikan.
JAMA, 2006 Jun 08; 295(21). PMID: 16757721
Highly Cited.
Mitosis detection in breast cancer histology images with deep neural networks.
Dan C Cireşan, Alessandro Giusti, Luca M Gambardella, Jürgen Schmidhuber.
Med Image Comput Comput Assist Interv, 2014 Mar 01; 16(Pt 2). PMID: 24579167
Highly Cited.
Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology.
Kaustav Bera, Kurt A Schalper, +2 authors, Anant Madabhushi.
Nat Rev Clin Oncol, 2019 Aug 11; 16(11). PMID: 31399699    Free PMC article.
Highly Cited. Review.
System-based approaches as prognostic tools for glioblastoma.
Manuela Salvucci, Zaitun Zakaria, +5 authors, Jochen H M Prehn.
BMC Cancer, 2019 Nov 14; 19(1). PMID: 31718568    Free PMC article.
A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival.
Mustafa I Jaber, Bing Song, +5 authors, Christopher W Szeto.
Breast Cancer Res, 2020 Jan 30; 22(1). PMID: 31992350    Free PMC article.
Spatially multiplexed RNA in situ hybridization to reveal tumor heterogeneity.
Lena Voith von Voithenberg, Anna Fomitcheva Khartchenko, +2 authors, Govind V Kaigala.
Nucleic Acids Res, 2019 Dec 20; 48(3). PMID: 31853536    Free PMC article.
Emerging role of deep learning-based artificial intelligence in tumor pathology.
Yahui Jiang, Meng Yang, +2 authors, Yan Sun.
Cancer Commun (Lond), 2020 Apr 12; 40(4). PMID: 32277744    Free PMC article.
Deep learned tissue "fingerprints" classify breast cancers by ER/PR/Her2 status from H&E images.
Rishi R Rawat, Itzel Ortega, +4 authors, David B Agus.
Sci Rep, 2020 May 01; 10(1). PMID: 32350370    Free PMC article.
Digital Analysis in Breast Imaging.
Giovanna Negrão de Figueiredo, Michael Ingrisch, Eva Maria Fallenberg.
Breast Care (Basel), 2019 Jul 19; 14(3). PMID: 31316312    Free PMC article.
Deep Learning With Sampling in Colon Cancer Histology.
Mary Shapcott, Katherine J Hewitt, Nasir Rajpoot.
Front Bioeng Biotechnol, 2019 Apr 12; 7. PMID: 30972333    Free PMC article.
Artificial intelligence in digital breast pathology: Techniques and applications.
Asmaa Ibrahim, Paul Gamble, +4 authors, Emad A Rakha.
Breast, 2020 Jan 15; 49. PMID: 31935669    Free PMC article.
Nonlinear multispectral imaging for tumor delineation.
Elena Beletkaia, Behdad Dashtbozorg, +2 authors, Herman L Offerhaus.
J Biomed Opt, 2020 Sep 05; 25(9). PMID: 32885620    Free PMC article.
Multi-Field-of-View Deep Learning Model Predicts Nonsmall Cell Lung Cancer Programmed Death-Ligand 1 Status from Whole-Slide Hematoxylin and Eosin Images.
Lingdao Sha, Boleslaw L Osinski, +7 authors, Stephen S F Yip.
J Pathol Inform, 2019 Sep 17; 10. PMID: 31523482    Free PMC article.
SHIFT: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning.
Erik A Burlingame, Mary McDonnell, +7 authors, Young Hwan Chang.
Sci Rep, 2020 Oct 17; 10(1). PMID: 33060677    Free PMC article.
Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains.
Nikhil Naik, Ali Madani, +5 authors, Richard Socher.
Nat Commun, 2020 Nov 18; 11(1). PMID: 33199723    Free PMC article.
Deep neural network models for computational histopathology: A survey.
Chetan L Srinidhi, Ozan Ciga, Anne L Martel.
Med Image Anal, 2020 Oct 14; 67. PMID: 33049577    Free PMC article.
Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images.
Javad Noorbakhsh, Saman Farahmand, +6 authors, Jeffrey H Chuang.
Nat Commun, 2020 Dec 15; 11(1). PMID: 33311458    Free PMC article.
Deep Learning Predicts Underlying Features on Pathology Images with Therapeutic Relevance for Breast and Gastric Cancer.
Renan Valieris, Lucas Amaro, +6 authors, Israel Tojal da Silva.
Cancers (Basel), 2020 Dec 16; 12(12). PMID: 33316873    Free PMC article.
Artificial Intelligence for Histology-Based Detection of Microsatellite Instability and Prediction of Response to Immunotherapy in Colorectal Cancer.
Lindsey A Hildebrand, Colin J Pierce, +2 authors, Asaf Maoz.
Cancers (Basel), 2021 Jan 27; 13(3). PMID: 33494280    Free PMC article.
Deep learning in cancer pathology: a new generation of clinical biomarkers.
Amelie Echle, Niklas Timon Rindtorff, +3 authors, Jakob Nikolas Kather.
Br J Cancer, 2020 Nov 19; 124(4). PMID: 33204028    Free PMC article.
Histologic grading of breast carcinoma: a multi-institution study of interobserver variation using virtual microscopy.
Paula S Ginter, Romana Idress, +6 authors, Malini Harigopal.
Mod Pathol, 2020 Oct 21; 34(4). PMID: 33077923    Free PMC article.
Integrating biology and access to care in addressing breast cancer disparities: 25 years' research experience in the Carolina Breast Cancer Study.
Marc A Emerson, Katherine E Reeder-Hayes, +6 authors, Melissa A Troester.
Curr Breast Cancer Rep, 2021 Apr 06; 12(3). PMID: 33815665    Free PMC article.
Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology.
Cesare Lancellotti, Pierandrea Cancian, +4 authors, Luca Di Tommaso.
Cells, 2021 May 01; 10(4). PMID: 33918173    Free PMC article.
Deep learning in histopathology: the path to the clinic.
Jeroen van der Laak, Geert Litjens, Francesco Ciompi.
Nat Med, 2021 May 16; 27(5). PMID: 33990804
Deep learning predicts gene expression as an intermediate data modality to identify susceptibility patterns in Mycobacterium tuberculosis infected Diversity Outbred mice.
Thomas E Tavolara, M K K Niazi, +3 authors, Metin N Gurcan.
EBioMedicine, 2021 May 18; 67. PMID: 34000621    Free PMC article.
Dual-scale categorization based deep learning to evaluate programmed cell death ligand 1 expression in non-small cell lung cancer.
Xiangyun Wang, Peilin Chen, +5 authors, Qiang Fu.
Medicine (Baltimore), 2021 May 21; 100(20). PMID: 34011092    Free PMC article.