Journal Article
. 2021 Feb; 23(2):e23436.
doi: 10.2196/23436.

Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study

Max Schmitt 1 Roman Christoph Maron 1 Achim Hekler 1 Albrecht Stenzinger 2 Axel Hauschild 3 Michael Weichenthal 3 Markus Tiemann 4 Dieter Krahl 5 Heinz Kutzner 6 Jochen Sven Utikal 7 Sebastian Haferkamp 8 Jakob Nikolas Kather 9 Frederick Klauschen 10 Eva Krieghoff-Henning 1 Stefan Fröhling 11 Christof von Kalle 12 Titus Josef Brinker 1 
Affiliations
  • PMID: 33528370
  •     23 References

Abstract

Background: An increasing number of studies within digital pathology show the potential of artificial intelligence (AI) to diagnose cancer using histological whole slide images, which requires large and diverse data sets. While diversification may result in more generalizable AI-based systems, it can also introduce hidden variables. If neural networks are able to distinguish/learn hidden variables, these variables can introduce batch effects that compromise the accuracy of classification systems.

Objective: The objective of the study was to analyze the learnability of an exemplary selection of hidden variables (patient age, slide preparation date, slide origin, and scanner type) that are commonly found in whole slide image data sets in digital pathology and could create batch effects.

Methods: We trained four separate convolutional neural networks (CNNs) to learn four variables using a data set of digitized whole slide melanoma images from five different institutes. For robustness, each CNN training and evaluation run was repeated multiple times, and a variable was only considered learnable if the lower bound of the 95% confidence interval of its mean balanced accuracy was above 50.0%.

Results: A mean balanced accuracy above 50.0% was achieved for all four tasks, even when considering the lower bound of the 95% confidence interval. Performance between tasks showed wide variation, ranging from 56.1% (slide preparation date) to 100% (slide origin).

Conclusions: Because all of the analyzed hidden variables are learnable, they have the potential to create batch effects in dermatopathology data sets, which negatively affect AI-based classification systems. Practitioners should be aware of these and similar pitfalls when developing and evaluating such systems and address these and potentially other batch effect variables in their data sets through sufficient data set stratification.

Keywords: artifacts; artificial intelligence; clinical pathology; convolutional neural networks; deep learning; digital pathology; machine learning; neural networks; pathology; pitfalls.

Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.
Nicolas Coudray, Paolo Santiago Ocampo, +6 authors, Aristotelis Tsirigos.
Nat Med, 2018 Sep 19; 24(10). PMID: 30224757
Highly Cited.
Stain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology.
Andrew Janowczyk, Ajay Basavanhally, Anant Madabhushi.
Comput Med Imaging Graph, 2016 Jul 05; 57. PMID: 27373749    Free PMC article.
Removing batch effects from histopathological images for enhanced cancer diagnosis.
Sonal Kothari, John H Phan, +3 authors, May D Wang.
IEEE J Biomed Health Inform, 2014 May 09; 18(3). PMID: 24808220    Free PMC article.
Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer.
Babak Ehteshami Bejnordi, Mitko Veta, +67 authors, Rui Venâncio.
JAMA, 2017 Dec 14; 318(22). PMID: 29234806    Free PMC article.
Highly Cited.
A study about color normalization methods for histopathology images.
Santanu Roy, Alok Kumar Jain, Shyam Lal, Jyoti Kini.
Micron, 2018 Aug 11; 114. PMID: 30096632
Review.
Multiscale integration of -omic, imaging, and clinical data in biomedical informatics.
John H Phan, Chang F Quo, Chihwen Cheng, May Dongmei Wang.
IEEE Rev Biomed Eng, 2012 Dec 13; 5. PMID: 23231990    Free PMC article.
Review.
Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology.
Kaustav Bera, Kurt A Schalper, +2 authors, Anant Madabhushi.
Nat Rev Clin Oncol, 2019 Aug 11; 16(11). PMID: 31399699    Free PMC article.
Highly Cited. Review.
Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.
John R Zech, Marcus A Badgeley, +3 authors, Eric Karl Oermann.
PLoS Med, 2018 Nov 07; 15(11). PMID: 30399157    Free PMC article.
Highly Cited.
Accurate and reproducible invasive breast cancer detection in whole-slide images: A Deep Learning approach for quantifying tumor extent.
Angel Cruz-Roa, Hannah Gilmore, +6 authors, Anant Madabhushi.
Sci Rep, 2017 Apr 19; 7. PMID: 28418027    Free PMC article.
Highly Cited.
Automatic batch-invariant color segmentation of histological cancer images.
Sonal Kothari, John H Phan, +5 authors, May D Wang.
Proc IEEE Int Symp Biomed Imaging, 2011 Mar 01; 2011. PMID: 27532016    Free PMC article.
Natural and sun-induced aging of human skin.
Laure Rittié, Gary J Fisher.
Cold Spring Harb Perspect Med, 2015 Jan 07; 5(1). PMID: 25561721    Free PMC article.
Highly Cited. Review.
Causability and explainability of artificial intelligence in medicine.
Andreas Holzinger, Georg Langs, +2 authors, Heimo Müller.
Wiley Interdiscip Rev Data Min Knowl Discov, 2020 Feb 25; 9(4). PMID: 32089788    Free PMC article.
Review.
HistoQC: An Open-Source Quality Control Tool for Digital Pathology Slides.
Andrew Janowczyk, Ren Zuo, +2 authors, Anant Madabhushi.
JCO Clin Cancer Inform, 2019 Apr 17; 3. PMID: 30990737    Free PMC article.
Scale normalization of histopathological images for batch invariant cancer diagnostic models.
Sonal Kothari, John H Phan, May D Wang.
Annu Int Conf IEEE Eng Med Biol Soc, 2013 Feb 01; 2012. PMID: 23366904    Free PMC article.
Artificial Intelligence and Digital Pathology: Challenges and Opportunities.
Hamid Reza Tizhoosh, Liron Pantanowitz.
J Pathol Inform, 2019 Jan 05; 9. PMID: 30607305    Free PMC article.
Highly Cited. Review.
Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer.
Jakob Nikolas Kather, Alexander T Pearson, +14 authors, Tom Luedde.
Nat Med, 2019 Jun 05; 25(7). PMID: 31160815    Free PMC article.
Highly Cited.
Pathologist-level classification of histopathological melanoma images with deep neural networks.
Achim Hekler, Jochen Sven Utikal, +10 authors, Titus Josef Brinker.
Eur J Cancer, 2019 May 28; 115. PMID: 31129383
Applications and challenges of digital pathology and whole slide imaging.
C Higgins.
Biotech Histochem, 2015 May 16; 90(5). PMID: 25978139
Review.
Image analysis and machine learning in digital pathology: Challenges and opportunities.
Anant Madabhushi, George Lee.
Med Image Anal, 2016 Jul 18; 33. PMID: 27423409    Free PMC article.
Highly Cited. Review.
Image processing in digital pathology: an opportunity to solve inter-batch variability of immunohistochemical staining.
Yves-Rémi Van Eycke, Justine Allard, +2 authors, Christine Decaestecker.
Sci Rep, 2017 Feb 22; 7. PMID: 28220842    Free PMC article.
Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images.
Achim Hekler, Jochen S Utikal, +12 authors, Titus J Brinker.
Eur J Cancer, 2019 Jul 22; 118. PMID: 31325876
A review of artifacts in histopathology.
Syed Ahmed Taqi, Syed Abdus Sami, Lateef Begum Sami, Syed Ahmed Zaki.
J Oral Maxillofac Pathol, 2018 Aug 31; 22(2). PMID: 30158787    Free PMC article.
Review.
Why Batch Effects Matter in Omics Data, and How to Avoid Them.
Wilson Wen Bin Goh, Wei Wang, Limsoon Wong.
Trends Biotechnol, 2017 Mar 30; 35(6). PMID: 28351613
Highly Cited. Review.