Dataset Information

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites.

ABSTRACT: Performance problems in applications should ideally be detected as soon as they occur, i.e., directly when the causing code modification is added to the code repository. To this end, complex and cost-intensive application benchmarks or lightweight but less relevant microbenchmarks can be added to existing build pipelines to ensure performance goals. In this paper, we show how the practical relevance of microbenchmark suites can be improved and verified based on the application flow during an application benchmark run. We propose an approach to determine the overlap of common function calls between application and microbenchmarks, describe a method which identifies redundant microbenchmarks, and present a recommendation algorithm which reveals relevant functions that are not covered by microbenchmarks yet. A microbenchmark suite optimized in this way can easily test all functions determined to be relevant by application benchmarks after every code change, thus, significantly reducing the risk of undetected performance problems. Our evaluation using two time series databases shows that, depending on the specific application scenario, application benchmarks cover different functions of the system under test. Their respective microbenchmark suites cover between 35.62% and 66.29% of the functions called during the application benchmark, offering substantial room for improvement. Through two use cases-removing redundancies in the microbenchmark suite and recommendation of yet uncovered functions-we decrease the total number of microbenchmarks and increase the practical relevance of both suites. Removing redundancies can significantly reduce the number of microbenchmarks (and thus the execution time as well) to ~10% and ~23% of the original microbenchmark suites, whereas recommendation identifies up to 26 and 14 newly, uncovered functions to benchmark to improve the relevance. By utilizing the differences and synergies of application benchmarks and microbenchmarks, our approach potentially enables effective software performance assurance with performance tests of multiple granularities.

SUBMITTER: Grambow M

PROVIDER: S-EPMC8176533 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundClassification is the problem of assigning each input object to one of a finite number of classes. This problem has been extensively studied in machine learning and statistics, and there are numerous applications to bioinformatics as well as many other fields. Building a multiclass classifier has been a challenge, where the direct approach of altering the binary classification algorithm to accommodate more than two classes can be computationally too expensive. Hence the indirect approach of using binary decomposition has been commonly used, in which retrieving the class posterior probabilities from the set of binary posterior probabilities given by the individual binary classifiers has been a major issue.MethodsIn this work, we present an extension of a recently introduced probabilistic kernel-based learning algorithm called the Classification Relevance Units Machine (CRUM) to the multiclass setting to increase its applicability. The extension is achieved under the error correcting output codes framework. The probabilistic outputs of the binary CRUMs are preserved using a proposed linear-time decoding algorithm, an alternative to the generalized Bradley-Terry (GBT) algorithm whose application to large-scale prediction settings is prohibited by its computational complexity. The resulting classifier is called the Multiclass Relevance Units Machine (McRUM).ResultsThe evaluation of McRUM on a variety of real small-scale benchmark datasets shows that our proposed Naïve decoding algorithm is computationally more efficient than the GBT algorithm while maintaining a similar level of predictive accuracy. Then a set of experiments on a larger scale dataset for small ncRNA classification have been conducted with Naïve McRUM and compared with the Gaussian and linear SVM. Although McRUM's predictive performance is slightly lower than the Gaussian SVM, the results show that the similar level of true positive rate can be achieved by sacrificing false positive rate slightly. Furthermore, McRUM is computationally more efficient than the SVM, which is an important factor for large-scale analysis.ConclusionsWe have proposed McRUM, a multiclass extension of binary CRUM. McRUM with Naïve decoding algorithm is computationally efficient in run-time and its predictive performance is comparable to the well-known SVM, showing its potential in solving large-scale multiclass problems in bioinformatics and other fields of study.

Project description:BackgroundInference of active regulatory cascades under specific molecular and environmental perturbations is a recurring task in transcriptional data analysis. Commercial tools based on large, manually curated networks of causal relationships offering such functionality have been used in thousands of articles in the biomedical literature. The adoption and extension of such methods in the academic community has been hampered by the lack of freely available, efficient algorithms and an accompanying demonstration of their applicability using current public networks.ResultsIn this article, we propose a new statistical method that will infer likely upstream regulators based on observed patterns of up- and down-regulated transcripts. The method is suitable for use with public interaction networks with a mix of signed and unsigned causal edges. It subsumes and extends two previously published approaches and we provide a novel algorithmic method for efficient statistical inference. Notably, we demonstrate the feasibility of using the approach to generate biological insights given current public networks in the context of controlled in-vitro overexpression experiments, stem-cell differentiation data and animal disease models. We also provide an efficient implementation of our method in the R package QuaternaryProd available to download from Bioconductor.ConclusionsIn this work, we have closed an important gap in utilizing causal networks to analyze differentially expressed genes. Our proposed Quaternary test statistic incorporates all available evidence on the potential relevance of an upstream regulator. The new approach broadens the use of these types of statistics for highly curated signed networks in which ambiguities arise but also enables the use of networks with unsigned edges. We design and implement a novel computational method that can efficiently estimate p-values for upstream regulators in current biological settings. We demonstrate the ready applicability of the implemented method to analyze differentially expressed genes using the publicly available networks.

Project description:The purpose of this study was to develop a new diagnostic technique for measuring bone mineral density (BMD) for the assessment of osteoporosis, which improves upon the coherent to Compton scattering ratio (CCSR) method, which was first developed in the 1980s. To help the authors achieve these goals, they have identified and studied two new indices for CCSR, the forward scattered to backward scattered (FS-BS) and the forward scattered to transmitted (FS-T) ratios. They believe that, at small angles, these two parameters can offer a practical in vivo determination of BMD that can be used to overcome the limitations of past CCSR systems, including high radiation dosages, costs, and examination durations.In previous CCSR studies, a high-activity radioactive source with a long half-live (usually (241)Am) and an expensive and bulky cryogenic HPGe detector were applied to both in vivo and in vitro measurements. To make this technique more suitable for clinical applications, the possibility of using a standard diagnostic x-ray tube generating a continuous spectrum was investigated in this paper. Scattered radiation from trabecular bone-simulating phantoms containing various mineral densities that span the normal range of in vivo BMD was collected in this study using relatively inexpensive noncryogenic CdTe or NaI detectors.The initial results demonstrate that a modified version of CCSR can be successfully applied to trabecular bone assessment using a diagnostic x-ray tube with a continuous spectrum in two variations, the FS-BS and the FS-T ratio. When FS-BS is measured, intensity spectra in the forward and backward directions must be collected while FS-T requires only the integral intensity of the scattered and transmitted (T) spectra in the energy region above 40 keV. For both of these methods, forward scattering angles less than or equal to 15° and backward scattering angles greater than or equal to (165°= 180° - 15°) are needed.The authors determined that FS-T is more sensitive to changes in BMD than transmission or absorption alone and that the FS-BS method can yield an absolute measurement of the mean atomic number of the scattering medium, after a correction for path-dependent attenuation. Since this study determined that the FS-T ratio is independent of the incident energy over a broad energy region, it will be possible to apply FS-T to bone densitometry using inexpensive integral photon detectors. The authors believe that, by replacing the radionuclide source with an x-ray tube and the cryogenically cooled HPGe detector with a single solid state CdTe, NaI, or silicon detector or an annular array of detectors, as suggested in this study, the past difficulties of CCSR concerning high radiation exposure, costs, and durations as well as lack of convenience can be overcome and that CCSR could eventually become popular in clinical settings.

Project description:BackgroundMachine learning (ML) is now widely deployed in our everyday lives. Building robust ML models requires a massive amount of data for training. Traditional ML algorithms require training data centralization, which raises privacy and data governance issues. Federated learning (FL) is an approach to overcome this issue. We focused on applying FL on vertically partitioned data, in which an individual's record is scattered among different sites.ObjectiveThe aim of this study was to perform FL on vertically partitioned data to achieve performance comparable to that of centralized models without exposing the raw data.MethodsWe used three different datasets (Adult income, Schwannoma, and eICU datasets) and vertically divided each dataset into different pieces. Following the vertical division of data, overcomplete autoencoder-based model training was performed for each site. Following training, each site's data were transformed into latent data, which were aggregated for training. A tabular neural network model with categorical embedding was used for training. A centrally based model was used as a baseline model, which was compared to that of FL in terms of accuracy and area under the receiver operating characteristic curve (AUROC).ResultsThe autoencoder-based network successfully transformed the original data into latent representations with no domain knowledge applied. These altered data were different from the original data in terms of the feature space and data distributions, indicating appropriate data security. The loss of performance was minimal when using an overcomplete autoencoder; accuracy loss was 1.2%, 8.89%, and 1.23%, and AUROC loss was 1.1%, 0%, and 1.12% in the Adult income, Schwannoma, and eICU dataset, respectively.ConclusionsWe proposed an autoencoder-based ML model for vertically incomplete data. Since our model is based on unsupervised learning, no domain-specific knowledge is required in individual sites. Under the circumstances where direct data sharing is not available, our approach may be a practical solution enabling both data protection and building a robust model.

Dataset Information

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets