Journal Article
. 2020 Aug;11(1).
doi: 10.1038/s41467-020-17678-4.

A deep learning model to predict RNA-Seq expression of tumours from whole slide images

Benoît Schmauch 1 Alberto Romagnoni 2 Elodie Pronier 2 Charlie Saillard 2 Pascale Maillé 3 Julien Calderaro 3 Aurélie Kamoun 2 Meriem Sefta 2 Sylvain Toldo 2 Mikhail Zaslavskiy 2 Thomas Clozel 2 Matahi Moarii 2 Pierre Courtiol 2 Gilles Wainrib 4 
  • PMID: 32747659
  •     59 References
  •     11 citations


Deep learning methods for digital pathology analysis are an effective way to address multiple clinical questions, from diagnosis to prediction of treatment outcomes. These methods have also been used to predict gene mutations from pathology images, but no comprehensive evaluation of their potential for extracting molecular features from histology slides has yet been performed. We show that HE2RNA, a model based on the integration of multiple data modes, can be trained to systematically predict RNA-Seq profiles from whole-slide images alone, without expert annotation. Through its interpretable design, HE2RNA provides virtual spatialization of gene expression, as validated by CD3- and CD20-staining on an independent dataset. The transcriptomic representation learned by HE2RNA can also be transferred on other datasets, even of small size, to increase prediction performance for specific molecular phenotypes. We illustrate the use of this approach in clinical diagnosis purposes such as the identification of tumors with microsatellite instability.

Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.
Nicolas Coudray, Paolo Santiago Ocampo, +6 authors, Aristotelis Tsirigos.
Nat Med, 2018 Sep 19; 24(10). PMID: 30224757
Highly Cited.
Cell-Type-Specific Gene Expression Profiling in Adult Mouse Brain Reveals Normal and Disease-State Signatures.
Nicolas Merienne, Cécile Meunier, +11 authors, Nicole Déglon.
Cell Rep, 2019 Feb 28; 26(9). PMID: 30811995
Molecular Link between Liver Fibrosis and Hepatocellular Carcinoma.
Toshiharu Sakurai, Masatoshi Kudo.
Liver Cancer, 2014 Jan 09; 2(3-4). PMID: 24400223    Free PMC article.
Correlating nuclear morphometric patterns with estrogen receptor status in breast cancer pathologic specimens.
Rishi R Rawat, Daniel Ruderman, +2 authors, David B Agus.
NPJ Breast Cancer, 2018 Sep 14; 4. PMID: 30211313    Free PMC article.
The cancer genome.
Michael R Stratton, Peter J Campbell, P Andrew Futreal.
Nature, 2009 Apr 11; 458(7239). PMID: 19360079    Free PMC article.
Highly Cited. Review.
Quantitative nuclear histomorphometry predicts oncotype DX risk categories for early stage ER+ breast cancer.
Jon Whitney, German Corredor, +6 authors, Anant Madabhushi.
BMC Cancer, 2018 Jun 01; 18(1). PMID: 29848291    Free PMC article.
Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples.
Riku Turkki, Nina Linder, +2 authors, Johan Lundin.
J Pathol Inform, 2016 Oct 01; 7. PMID: 27688929    Free PMC article.
A National Cancer Institute Workshop on Microsatellite Instability for cancer detection and familial predisposition: development of international criteria for the determination of microsatellite instability in colorectal cancer.
C R Boland, S N Thibodeau, +8 authors, S Srivastava.
Cancer Res, 1998 Nov 21; 58(22). PMID: 9823339
Highly Cited. Review.
Ki-67 antigen expression in hepatocellular carcinoma using monoclonal antibody MIB1. A comparison with proliferating cell nuclear antigen.
I O Ng, J Na, +2 authors, M Ng.
Am J Clin Pathol, 1995 Sep 01; 104(3). PMID: 7545866
Hallmarks of cancer: the next generation.
Douglas Hanahan, Robert A Weinberg.
Cell, 2011 Mar 08; 144(5). PMID: 21376230
Highly Cited. Review.
Small hepatocellular carcinoma of single nodular type: a specific reference to its surrounding cancerous area undetected radiologically and macroscopically.
T Maeda, K Takenaka, +5 authors, M Tsuneyoshi.
J Surg Oncol, 1995 Oct 01; 60(2). PMID: 7564384
Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer.
Jakob Nikolas Kather, Alexander T Pearson, +14 authors, Tom Luedde.
Nat Med, 2019 Jun 05; 25(7). PMID: 31160815    Free PMC article.
Highly Cited.
Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard.
Wouter Bulten, Péter Bándi, +7 authors, Geert Litjens.
Sci Rep, 2019 Jan 31; 9(1). PMID: 30696866    Free PMC article.
SLIC superpixels compared to state-of-the-art superpixel methods.
Radhakrishna Achanta, Appu Shaji, +3 authors, Sabine Süsstrunk.
IEEE Trans Pattern Anal Mach Intell, 2012 May 30; 34(11). PMID: 22641706
Highly Cited.
Next-generation sequencing: advances and applications in cancer diagnosis.
Simona Serratì, Simona De Summa, +4 authors, Rosamaria Pinto.
Onco Targets Ther, 2016 Dec 17; 9. PMID: 27980425    Free PMC article.
And They Said It Couldn't Be Done: Predicting Known Driver Mutations From H&E Slides.
Michael C Montalto, Robin Edwards.
J Pathol Inform, 2019 Jun 01; 10. PMID: 31149368    Free PMC article.
Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification.
Le Hou, Dimitris Samaras, +3 authors, Joel H Saltz.
Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, 2016 Nov 01; 2016. PMID: 27795661    Free PMC article.
Highly Cited.
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2.
Michael I Love, Wolfgang Huber, Simon Anders.
Genome Biol, 2014 Dec 18; 15(12). PMID: 25516281    Free PMC article.
Highly Cited.
Cyclins and breast cancer.
Robert L Sutherland, Elizabeth A Musgrove.
J Mammary Gland Biol Neoplasia, 2004 Apr 15; 9(1). PMID: 15082921
A Practical Guide to Whole Slide Imaging: A White Paper From the Digital Pathology Association.
Mark D Zarella, Douglas Bowman, +6 authors, Douglas J Hartman.
Arch Pathol Lab Med, 2018 Oct 12; 143(2). PMID: 30307746
Highly Cited. Review.
Current perspectives on CHEK2 mutations in breast cancer.
Panagiotis Apostolou, Ioannis Papasotiriou.
Breast Cancer (Dove Med Press), 2017 May 30; 9. PMID: 28553140    Free PMC article.
Characterization of GMP-17, a granule membrane protein that moves to the plasma membrane of natural killer cells following target cell recognition.
Q G Medley, N Kedersha, +4 authors, P Anderson.
Proc Natl Acad Sci U S A, 1996 Jan 23; 93(2). PMID: 8570616    Free PMC article.
The tetraspanin CD53 modulates responses from activating NK cell receptors, promoting LFA-1 activation and dampening NK cell effector functions.
Izabela Todros-Dawda, Lise Kveberg, John T Vaage, Marit Inngjerdingen.
PLoS One, 2014 May 17; 9(5). PMID: 24832104    Free PMC article.
Prognosis of hepatocellular carcinoma: the BCLC staging classification.
J M Llovet, C Brú, J Bruix.
Semin Liver Dis, 1999 Oct 13; 19(3). PMID: 10518312
Highly Cited.
The prognostic landscape of genes and infiltrating immune cells across human cancers.
Andrew J Gentles, Aaron M Newman, +11 authors, Ash A Alizadeh.
Nat Med, 2015 Jul 21; 21(8). PMID: 26193342    Free PMC article.
Highly Cited.
Systematic exploration of cell morphological phenotypes associated with a transcriptomic query.
Isar Nassiri, Matthew N McCall.
Nucleic Acids Res, 2018 Jul 17; 46(19). PMID: 30011038    Free PMC article.
Predicting cancer outcomes from histology and genomics using convolutional networks.
Pooya Mobadersany, Safoora Yousefi, +5 authors, Lee A D Cooper.
Proc Natl Acad Sci U S A, 2018 Mar 14; 115(13). PMID: 29531073    Free PMC article.
Highly Cited.
QuPath: Open source software for digital pathology image analysis.
Peter Bankhead, Maurice B Loughrey, +10 authors, Peter W Hamilton.
Sci Rep, 2017 Dec 06; 7(1). PMID: 29203879    Free PMC article.
Highly Cited.
Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features.
Haibo Wang, Angel Cruz-Roa, +6 authors, Anant Madabhushi.
J Med Imaging (Bellingham), 2015 Jul 15; 1(3). PMID: 26158062    Free PMC article.
Identification and sequence of a fourth human T cell antigen receptor chain.
E Y Loh, L L Lanier, +4 authors, A Weiss.
Nature, 1987 Dec 10; 330(6148). PMID: 2825032
Predicting Survival After Hepatocellular Carcinoma Resection Using Deep Learning on Histological Slides.
Charlie Saillard, Benoit Schmauch, +16 authors, Julien Calderaro.
Hepatology, 2020 Feb 29; 72(6). PMID: 32108950
Targeting the Complement Pathway as a Therapeutic Strategy in Lung Cancer.
Emily K Kleczko, Jeff W Kwak, Erin L Schenk, Raphael A Nemenoff.
Front Immunol, 2019 May 28; 10. PMID: 31134065    Free PMC article.
Differential expression analysis for sequence count data.
Simon Anders, Wolfgang Huber.
Genome Biol, 2010 Oct 29; 11(10). PMID: 20979621    Free PMC article.
Highly Cited.
Deep-Learning Convolutional Neural Networks Accurately Classify Genetic Mutations in Gliomas.
P Chang, J Grinband, +11 authors, D Chow.
AJNR Am J Neuroradiol, 2018 May 12; 39(7). PMID: 29748206    Free PMC article.
Highly Cited.
Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study.
Wouter Bulten, Hans Pinckaers, +6 authors, Geert Litjens.
Lancet Oncol, 2020 Jan 14; 21(2). PMID: 31926805
Microsatellite instability in colorectal cancer.
C Richard Boland, Ajay Goel.
Gastroenterology, 2010 Apr 28; 138(6). PMID: 20420947    Free PMC article.
Highly Cited. Review.
Whole Slide Imaging Versus Microscopy for Primary Diagnosis in Surgical Pathology: A Multicenter Blinded Randomized Noninferiority Study of 1992 Cases (Pivotal Study).
Sanjay Mukhopadhyay, Michael D Feldman, +31 authors, Clive R Taylor.
Am J Surg Pathol, 2017 Sep 30; 42(1). PMID: 28961557    Free PMC article.
Is there a difference between T- and B-lymphocyte morphology?
Dmitry I Strokotov, Maxim A Yurkin, +4 authors, Valeri P Maltsev.
J Biomed Opt, 2010 Jan 12; 14(6). PMID: 20059274
Clinicopathological and prognostic significance of high Ki-67 labeling index in hepatocellular carcinoma patients: a meta-analysis.
Yihuan Luo, Fanghui Ren, +5 authors, Gang Chen.
Int J Clin Exp Med, 2015 Sep 18; 8(7). PMID: 26379815    Free PMC article.
Next-Generation Sequencing in Oncology: Genetic Diagnosis, Risk Prediction and Cancer Classification.
Rick Kamps, Rita D Brandão, +4 authors, Andrea Romano.
Int J Mol Sci, 2017 Feb 02; 18(2). PMID: 28146134    Free PMC article.
Highly Cited. Review.
Association of csk-homologous kinase (CHK) (formerly MATK) with HER-2/ErbB-2 in breast cancer cells.
S Zrihan-Licht, J Lim, +3 authors, H Avraham.
J Biol Chem, 1997 Jan 17; 272(3). PMID: 8999872
Integrated analysis of transcriptomic and proteomic data.
Saad Haider, Ranadip Pal.
Curr Genomics, 2013 Oct 02; 14(2). PMID: 24082820    Free PMC article.
Highly Cited.
A long-term survivor of ruptured hepatocellular carcinoma after hepatic resection.
K Shirabe, M Kitamura, +3 authors, K Sugimachi.
J Gastroenterol Hepatol, 1995 May 01; 10(3). PMID: 7548817
TCGAbiolinks: an R/Bioconductor package for integrative analysis of TCGA data.
Antonio Colaprico, Tiago C Silva, +10 authors, Houtan Noushmehr.
Nucleic Acids Res, 2015 Dec 26; 44(8). PMID: 26704973    Free PMC article.
Highly Cited.
Colorectal and other cancer risks for carriers and noncarriers from families with a DNA mismatch repair gene mutation: a prospective cohort study.
Aung Ko Win, Joanne P Young, +22 authors, Mark A Jenkins.
J Clin Oncol, 2012 Feb 15; 30(9). PMID: 22331944    Free PMC article.
Highly Cited.
Comprehensive molecular characterization of human colon and rectal cancer.
Cancer Genome Atlas Network.
Nature, 2012 Jul 20; 487(7407). PMID: 22810696    Free PMC article.
Highly Cited.
Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning.
Yair Rivenson, Hongda Wang, +10 authors, Aydogan Ozcan.
Nat Biomed Eng, 2019 May 31; 3(6). PMID: 31142829
Highly Cited.
Genomics and emerging biomarkers for immunotherapy of colorectal cancer.
Jakob Nikolas Kather, Niels Halama, Dirk Jaeger.
Semin Cancer Biol, 2018 Mar 05; 52(Pt 2). PMID: 29501787
First FDA Approval Agnostic of Cancer Site - When a Biomarker Defines the Indication.
Steven Lemery, Patricia Keegan, Richard Pazdur.
N Engl J Med, 2017 Oct 12; 377(15). PMID: 29020592
Highly Cited.
Systematic analysis of breast cancer morphology uncovers stromal features associated with survival.
Andrew H Beck, Ankur R Sangoi, +6 authors, Daphne Koller.
Sci Transl Med, 2011 Nov 11; 3(108). PMID: 22072638
Highly Cited.
Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study.
Jakob Nikolas Kather, Johannes Krisam, +15 authors, Niels Halama.
PLoS Med, 2019 Jan 25; 16(1). PMID: 30677016    Free PMC article.
Highly Cited.
A survey of best practices for RNA-seq data analysis.
Ana Conesa, Pedro Madrigal, +8 authors, Ali Mortazavi.
Genome Biol, 2016 Jan 28; 17. PMID: 26813401    Free PMC article.
Highly Cited. Review.
A molecular portrait of microsatellite instability across multiple cancers.
Isidro Cortes-Ciriano, Sejoon Lee, +2 authors, Peter J Park.
Nat Commun, 2017 Jun 07; 8. PMID: 28585546    Free PMC article.
Highly Cited.
Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images.
Joel Saltz, Rajarsi Gupta, +14 authors, Vésteinn Thorsson.
Cell Rep, 2018 Apr 05; 23(1). PMID: 29617659    Free PMC article.
Highly Cited.
Mismatch repair deficiency predicts response of solid tumors to PD-1 blockade.
Dung T Le, Jennifer N Durham, +43 authors, Luis A Diaz.
Science, 2017 Jun 10; 357(6349). PMID: 28596308    Free PMC article.
Highly Cited.
CD19 and CD20 targeted vectors induce minimal activation of resting B lymphocytes.
Sabrina Kneissl, Qi Zhou, +3 authors, Christian J Buchholz.
PLoS One, 2013 Nov 19; 8(11). PMID: 24244415    Free PMC article.
From signatures to models: understanding cancer using microarrays.
Eran Segal, Nir Friedman, +2 authors, Daphne Koller.
Nat Genet, 2005 May 28; 37 Suppl. PMID: 15920529
Highly Cited. Review.
Genomics and the continuum of cancer care.
Ultan McDermott, James R Downing, Michael R Stratton.
N Engl J Med, 2011 Jan 28; 364(4). PMID: 21268726
Highly Cited. Review.
Array of hope.
E S Lander.
Nat Genet, 1999 Jan 23; 21(1 Suppl). PMID: 9915492
Through Predictive Personalized Medicine.
Giuseppe Giglia, Giuditta Gambino, Pierangelo Sardo.
Brain Sci, 2020 Sep 03; 10(9). PMID: 32872094    Free PMC article.
Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer.
Alona Levy-Jurgenson, Xavier Tekpli, Vessela N Kristensen, Zohar Yakhini.
Sci Rep, 2020 Nov 04; 10(1). PMID: 33139755    Free PMC article.
Reimagining T Staging Through Artificial Intelligence and Machine Learning Image Processing Approaches in Digital Pathology.
Kaustav Bera, Ian Katz, Anant Madabhushi.
JCO Clin Cancer Inform, 2020 Nov 10; 4. PMID: 33166198    Free PMC article.
How Do Machines Learn? Artificial Intelligence as a New Era in Medicine.
Oliwia Koteluk, Adrian Wartecki, +2 authors, Andrzej Mackiewicz.
J Pers Med, 2021 Jan 13; 11(1). PMID: 33430240    Free PMC article.
Artificial Intelligence for Histology-Based Detection of Microsatellite Instability and Prediction of Response to Immunotherapy in Colorectal Cancer.
Lindsey A Hildebrand, Colin J Pierce, +2 authors, Asaf Maoz.
Cancers (Basel), 2021 Jan 27; 13(3). PMID: 33494280    Free PMC article.
Deep learning framework for subject-independent emotion detection using wireless signals.
Ahsan Noor Khan, Achintha Avin Ihalage, +3 authors, Yang Hao.
PLoS One, 2021 Feb 04; 16(2). PMID: 33534826    Free PMC article.
QuPath: The global impact of an open source digital pathology system.
M P Humphries, P Maxwell, M Salto-Tellez.
Comput Struct Biotechnol J, 2021 Feb 19; 19. PMID: 33598100    Free PMC article.
Deep learning in cancer pathology: a new generation of clinical biomarkers.
Amelie Echle, Niklas Timon Rindtorff, +3 authors, Jakob Nikolas Kather.
Br J Cancer, 2020 Nov 19; 124(4). PMID: 33204028    Free PMC article.
The Ethics of Artificial Intelligence in Pathology and Laboratory Medicine: Principles and Practice.
Brian R Jackson, Ye Ye, +5 authors, Liron Pantanowitz.
Acad Pathol, 2021 Mar 02; 8. PMID: 33644301    Free PMC article.
Morphological Heterogeneity in Pancreatic Cancer Reflects Structural and Functional Divergence.
Petra Sántha, Daniela Lenggenhager, +5 authors, Caroline Verbeke.
Cancers (Basel), 2021 Mar 07; 13(4). PMID: 33672734    Free PMC article.
Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes.
James A Diao, Jason K Wang, +19 authors, Amaro Taylor-Weiner.
Nat Commun, 2021 Mar 14; 12(1). PMID: 33712588    Free PMC article.