Dataset Information

Neural network interpretation using descrambler groups.

ABSTRACT: The lack of interpretability and trust is a much-criticized feature of deep neural networks. In fully connected nets, the signaling between inner layers is scrambled because backpropagation training does not require perceptrons to be arranged in any particular order. The result is a black box; this problem is particularly severe in scientific computing and digital signal processing (DSP), where neural nets perform abstract mathematical transformations that do not reduce to features or concepts. We present here a group-theoretical procedure that attempts to bring inner-layer signaling into a human-readable form, the assumption being that this form exists and has identifiable and quantifiable features-for example, smoothness or locality. We applied the proposed method to DEERNet (a DSP network used in electron spin resonance) and managed to descramble it. We found considerable internal sophistication: the network spontaneously invents a bandpass filter, a notch filter, a frequency axis rescaling transformation, frequency-division multiplexing, group embedding, spectral filtering regularization, and a map from harmonic functions into Chebyshev polynomials-in 10 min of unattended training from a random initial guess.

SUBMITTER: Amey JL

PROVIDER: S-EPMC7865153 | biostudies-literature | 2021 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Neural network interpretation using descrambler groups.

Amey Jake L JL Keeley Jake J Choudhury Tajwar T Kuprov Ilya I

Proceedings of the National Academy of Sciences of the United States of America 20210201 5

The lack of interpretability and trust is a much-criticized feature of deep neural networks. In fully connected nets, the signaling between inner layers is scrambled because backpropagation training does not require perceptrons to be arranged in any particular order. The result is a black box; this problem is particularly severe in scientific computing and digital signal processing (DSP), where neural nets perform abstract mathematical transformations that do not reduce to features or concepts. ...[more]

PMID: 33500352

Similar Datasets

Project description:We developed a method to apply artificial neural networks (ANNs) for predicting time-series pharmacokinetics (PKs), and an interpretable the ANN-PK model, which can explain the evidence of prediction by applying Shapley additive explanations (SHAP). A previous population PK (PopPK) model of cyclosporin A was used as the comparison model. The patients' data were used for the ANN-PK model input, and the output by ANN was the clearance (CL). The estimated CL value from the ANN were substituted into the one-compartment with one-order absorption model, the concentrations were calculated, and the parameters of ANN were updated by the back-propagation method. Kernel SHAP was applied to the trained model and the SHAP value of each input was calculated. The root mean squared error for the PopPK model and the ANN-PK model were 41.1 and 31.0 ng/ml, respectively. The goodness of fit plots for the ANN-PK model represented more convergence to y = x compared with that for the PopPK model, with good model performance for the ANN-PK model. The most influential factors on CL output were age and body weight from the evaluation using Kernel SHAP, and these factors were incorporated into the PopPK model as the significant covariates of CL. The ANN-PK model could handle time-series data and showed higher prediction accuracy then the conventional PopPK model, and the scientific validity for the model could be evaluated by applying SHAP. Study Highlights WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC? A black-box property of an artificial neural network (ANN) decreases the scientific confidence of the model, and making it difficult to utilize the ANN in the medical field. Moreover, difficulty in handling the time-series data is a significant problem for applying the ANN for pharmacometrics study. WHAT QUESTION DID THIS STUDY ADDRESS? How can we apply the ANN for predicting the time-series pharmacokinetics (PKs) , and confirm the scientific validity of the ANN model? WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE? Using the ANN in combination with a conventional compartment (ANN-PK) model enabled to handle the time-series PK data, and the predicting performance of the model was higher than that of the population PK model. Furthermore, we could evaluate the scientific validity of the ANN model by applying the Shapley additive explanations. HOW MIGHT THIS CHANGE DRUG DISCOVERY, DEVELOPMENT, AND/OR THERAPEUTICS? We expect that our study will contribute to develop the interpretable ANN model, which can predict the time-series PKs, drug efficacies, and side effects with high prediction performance.

Project description:OBJECTIVES:To develop a proof-of-concept "interpretable" deep learning prototype that justifies aspects of its predictions from a pre-trained hepatic lesion classifier. METHODS:A convolutional neural network (CNN) was engineered and trained to classify six hepatic tumor entities using 494 lesions on multi-phasic MRI, described in Part 1. A subset of each lesion class was labeled with up to four key imaging features per lesion. A post hoc algorithm inferred the presence of these features in a test set of 60 lesions by analyzing activation patterns of the pre-trained CNN model. Feature maps were generated that highlight regions in the original image that correspond to particular features. Additionally, relevance scores were assigned to each identified feature, denoting the relative contribution of a feature to the predicted lesion classification. RESULTS:The interpretable deep learning system achieved 76.5% positive predictive value and 82.9% sensitivity in identifying the correct radiological features present in each test lesion. The model misclassified 12% of lesions. Incorrect features were found more often in misclassified lesions than correctly identified lesions (60.4% vs. 85.6%). Feature maps were consistent with original image voxels contributing to each imaging feature. Feature relevance scores tended to reflect the most prominent imaging criteria for each class. CONCLUSIONS:This interpretable deep learning system demonstrates proof of principle for illuminating portions of a pre-trained deep neural network's decision-making, by analyzing inner layers and automatically describing features contributing to predictions. KEY POINTS:• An interpretable deep learning system prototype can explain aspects of its decision-making by identifying relevant imaging features and showing where these features are found on an image, facilitating clinical translation. • By providing feedback on the importance of various radiological features in performing differential diagnosis, interpretable deep learning systems have the potential to interface with standardized reporting systems such as LI-RADS, validating ancillary features and improving clinical practicality. • An interpretable deep learning system could potentially add quantitative data to radiologic reports and serve radiologists with evidence-based decision support.

Project description:ImportanceMillions of clinicians rely daily on automated preliminary electrocardiogram (ECG) interpretation. Critical comparisons of machine learning-based automated analysis against clinically accepted standards of care are lacking.ObjectiveTo use readily available 12-lead ECG data to train and apply an explainability technique to a convolutional neural network (CNN) that achieves high performance against clinical standards of care.Design, setting, and participantsThis cross-sectional study was conducted using data from January 1, 2003, to December 31, 2018. Data were obtained in a commonly available 12-lead ECG format from a single-center tertiary care institution. All patients aged 18 years or older who received ECGs at the University of California, San Francisco, were included, yielding a total of 365 009 patients. Data were analyzed from January 1, 2019, to March 2, 2021.ExposuresA CNN was trained to predict the presence of 38 diagnostic classes in 5 categories from 12-lead ECG data. A CNN explainability technique called LIME (Linear Interpretable Model-Agnostic Explanations) was used to visualize ECG segments contributing to CNN diagnoses.Main outcomes and measuresArea under the receiver operating characteristic curve (AUC), sensitivity, and specificity were calculated for the CNN in the holdout test data set against cardiologist clinical diagnoses. For a second validation, 3 electrophysiologists provided consensus committee diagnoses against which the CNN, cardiologist clinical diagnosis, and MUSE (GE Healthcare) automated analysis performance was compared using the F1 score; AUC, sensitivity, and specificity were also calculated for the CNN against the consensus committee.ResultsA total of 992 748 ECGs from 365 009 adult patients (mean [SD] age, 56.2 [17.6] years; 183 600 women [50.3%]; and 175 277 White patients [48.0%]) were included in the analysis. In 91 440 test data set ECGs, the CNN demonstrated an AUC of at least 0.960 for 32 of 38 classes (84.2%). Against the consensus committee diagnoses, the CNN had higher frequency-weighted mean F1 scores than both cardiologists and MUSE in all 5 categories (CNN frequency-weighted F1 score for rhythm, 0.812; conduction, 0.729; chamber diagnosis, 0.598; infarct, 0.674; and other diagnosis, 0.875). For 32 of 38 classes (84.2%), the CNN had AUCs of at least 0.910 and demonstrated comparable F1 scores and higher sensitivity than cardiologists, except for atrial fibrillation (CNN F1 score, 0.847 vs cardiologist F1 score, 0.881), junctional rhythm (0.526 vs 0.727), premature ventricular complex (0.786 vs 0.800), and Wolff-Parkinson-White (0.800 vs 0.842). Compared with MUSE, the CNN had higher F1 scores for all classes except supraventricular tachycardia (CNN F1 score, 0.696 vs MUSE F1 score, 0.714). The LIME technique highlighted physiologically relevant ECG segments.Conclusions and relevanceThe results of this cross-sectional study suggest that readily available ECG data can be used to train a CNN algorithm to achieve comparable performance to clinical cardiologists and exceed the performance of MUSE automated analysis for most diagnoses, with some exceptions. The LIME explainability technique applied to CNNs highlights physiologically relevant ECG segments that contribute to the CNN's diagnoses.

Dataset Information

Neural network interpretation using descrambler groups.

Publications

Neural network interpretation using descrambler groups.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets