GhostiPy: An Efficient Signal Processing and Spectral Analysis Toolbox for Large Data.
Ontology highlight
ABSTRACT:
SUBMITTER: Chu JP
PROVIDER: S-EPMC8641918 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:A typical diffraction experiment will generate many images and data sets from different crystals in a very short time. This creates a challenge for the high-throughput operation of modern synchrotron beamlines as well as for the subsequent data processing. Novice users in particular may feel overwhelmed by the tables, plots and numbers that the different data-processing programs and software packages present to them. Here, some of the more common problems that a user has to deal with when processing a set of images that will finally make up a processed data set are shown, concentrating on difficulties that may often show up during the first steps along the path of turning the experiment (i.e. data collection) into a model (i.e. interpreted electron density). Difficulties such as unexpected crystal forms, issues in crystal handling and suboptimal choices of data-collection strategies can often be dealt with, or at least diagnosed, by analysing specific data characteristics during processing. In the end, one wants to distinguish problems over which one has no immediate control once the experiment is finished from problems that can be remedied a posteriori. A new software package, autoPROC, is also presented that combines third-party processing programs with new tools and an automated workflow script that is intended to provide users with both guidance and insight into the offline processing of data affected by the difficulties mentioned above, with particular emphasis on the automated treatment of multi-sweep data sets collected on multi-axis goniostats.
| S-EPMC3069744 | biostudies-literature
Project description:BackgroundIn a typical electrophysiological experiment, especially one that includes studying animal behavior, the data collected normally contain spikes, local field potentials, behavioral responses and other associated data. In order to obtain informative results, the data must be analyzed simultaneously with the experimental settings. However, most open-source toolboxes currently available for data analysis were developed to handle only a portion of the data and did not take into account the sorting of experimental conditions. Additionally, these toolboxes require that the input data be in a specific format, which can be inconvenient to users. Therefore, the development of a highly integrated toolbox that can process multiple types of data regardless of input data format and perform basic analysis for general electrophysiological experiments is incredibly useful.ResultsHere, we report the development of a Python based open-source toolbox, referred to as NeoAnalysis, to be used for quick electrophysiological data processing and analysis. The toolbox can import data from different data acquisition systems regardless of their formats and automatically combine different types of data into a single file with a standardized format. In cases where additional spike sorting is needed, NeoAnalysis provides a module to perform efficient offline sorting with a user-friendly interface. Then, NeoAnalysis can perform regular analog signal processing, spike train, and local field potentials analysis, behavioral response (e.g. saccade) detection and extraction, with several options available for data plotting and statistics. Particularly, it can automatically generate sorted results without requiring users to manually sort data beforehand. In addition, NeoAnalysis can organize all of the relevant data into an informative table on a trial-by-trial basis for data visualization. Finally, NeoAnalysis supports analysis at the population level.ConclusionsWith the multitude of general-purpose functions provided by NeoAnalysis, users can easily obtain publication-quality figures without writing complex codes. NeoAnalysis is a powerful and valuable toolbox for users doing electrophysiological experiments.
| S-EPMC5683334 | biostudies-literature
Project description:Real-time tracking of a waveform frequency content is essential for detection and analysis of fast rare events in communications, radar, radio astronomy, spectroscopy, sensing etc. This requires a method that can provide real-time spectrum analysis (RT-SA) of high-speed waveforms in a continuous and gap-free fashion. Digital signal processing is inefficient to perform RT-SA over instantaneous frequency bandwidths above the sub-GHz range and/or to track spectral changes faster than a few microseconds. Analog dispersion-induced frequency-to-time mapping enables RT-SA of short isolated pulse-like signals but cannot be extended to continuous waveforms. Here, we propose a universal analog processing approach for time-mapping a gap-free spectrogram -the prime method for dynamic frequency analysis- of an incoming arbitrary waveform, based on a simple sampling and dispersive delay scheme. In experiments, the spectrograms of GHz-bandwidth microwave signals are captured at a speed of ~5×109 Fourier transforms per second, allowing to intercept nanosecond-duration frequency transients in real time. This method opens new opportunities for dynamic frequency analysis and processing of high-speed waveforms.
| S-EPMC7335167 | biostudies-literature
Project description:BackgroundSeveral mathematical and statistical methods have been proposed in the last few years to analyze microarray data. Most of those methods involve complicated formulas, and software implementations that require advanced computer programming skills. Researchers from other areas may experience difficulties when they attempting to use those methods in their research. Here we present an user-friendly toolbox which allows large-scale gene expression analysis to be carried out by biomedical researchers with limited programming skills.ResultsHere, we introduce an user-friendly toolbox called GEDI (Gene Expression Data Interpreter), an extensible, open-source, and freely-available tool that we believe will be useful to a wide range of laboratories, and to researchers with no background in Mathematics and Computer Science, allowing them to analyze their own data by applying both classical and advanced approaches developed and recently published by Fujita et al.ConclusionGEDI is an integrated user-friendly viewer that combines the state of the art SVR, DVAR and SVAR algorithms, previously developed by us. It facilitates the application of SVR, DVAR and SVAR, further than the mathematical formulas present in the corresponding publications, and allows one to better understand the results by means of available visualizations. Both running the statistical methods and visualizing the results are carried out within the graphical user interface, rendering these algorithms accessible to the broad community of researchers in Molecular Biology.
| S-EPMC2194737 | biostudies-literature
Project description:BackgroundSpectral processing and post-experimental data analysis are the major tasks in NMR-based metabonomics studies. While there are commercial and free licensed software tools available to assist these tasks, researchers usually have to use multiple software packages for their studies because software packages generally focus on specific tasks. It would be beneficial to have a highly integrated platform, in which these tasks can be completed within one package. Moreover, with open source architecture, newly proposed algorithms or methods for spectral processing and data analysis can be implemented much more easily and accessed freely by the public.ResultsIn this paper, we report an open source software tool, Automics, which is specifically designed for NMR-based metabonomics studies. Automics is a highly integrated platform that provides functions covering almost all the stages of NMR-based metabonomics studies. Automics provides high throughput automatic modules with most recently proposed algorithms and powerful manual modules for 1D NMR spectral processing. In addition to spectral processing functions, powerful features for data organization, data pre-processing, and data analysis have been implemented. Nine statistical methods can be applied to analyses including: feature selection (Fisher's criterion), data reduction (PCA, LDA, ULDA), unsupervised clustering (K-Mean) and supervised regression and classification (PLS/PLS-DA, KNN, SIMCA, SVM). Moreover, Automics has a user-friendly graphical interface for visualizing NMR spectra and data analysis results. The functional ability of Automics is demonstrated with an analysis of a type 2 diabetes metabolic profile.ConclusionAutomics facilitates high throughput 1D NMR spectral processing and high dimensional data analysis for NMR-based metabonomics applications. Using Automics, users can complete spectral processing and data analysis within one software package in most cases. Moreover, with its open source architecture, interested researchers can further develop and extend this software based on the existing infrastructure.
| S-EPMC2666662 | biostudies-literature
Project description:BackgroundSpatial sequencing methods increasingly gain popularity within RNA biology studies. State-of-the-art techniques quantify messenger RNA expression levels from tissue sections and at the same time register information about the original locations of the molecules in the tissue. The resulting data sets are processed and analyzed by accompanying software that, however, is incompatible across inputs from different technologies.FindingsHere, we present spacemake, a modular, robust, and scalable spatial transcriptomics pipeline built in Snakemake and Python. Spacemake is designed to handle all major spatial transcriptomics data sets and can be readily configured for other technologies. It can process and analyze several samples in parallel, even if they stem from different experimental methods. Spacemake's unified framework enables reproducible data processing from raw sequencing data to automatically generated downstream analysis reports. Spacemake is built with a modular design and offers additional functionality such as sample merging, saturation analysis, and analysis of long reads as separate modules. Moreover, spacemake employs novoSpaRc to integrate spatial and single-cell transcriptomics data, resulting in increased gene counts for the spatial data set. Spacemake is open source and extendable, and it can be seamlessly integrated with existing computational workflows.
| S-EPMC9295369 | biostudies-literature
Project description:Nonlinear optical signal processing (NOSP) has the potential to significantly improve the throughput, flexibility, and cost-efficiency of optical communication networks by exploiting the intrinsically ultrafast optical nonlinear wave mixing. It can support digital signal processing speeds of up to terabits per second, far exceeding the line rate of the electronic counterpart. In NOSP, high-intensity light fields are used to generate nonlinear optical responses, which can be used to process optical signals. Great efforts have been devoted to developing new materials and structures for NOSP. However, one of the challenges in implementing NOSP is the requirement of high-intensity light fields, which is difficult to generate and maintain. This has been a major roadblock to realize practical NOSP systems for high-speed, high-capacity optical communications. Here, we propose using a parity-time (PT) symmetric microresonator system to significantly enhance the light intensity and support high-speed operation by relieving the bandwidth-efficiency limit imposed on conventional single resonator systems. The design concept is the co-existence of a PT symmetry broken regime for a narrow-linewidth pump wave and near-exceptional point operation for broadband signal and idler waves. This enables us to achieve a new NOSP system with two orders of magnitude improvement in efficiency compared to a single resonator. With a highly nonlinear AlGaAs-on-Insulator platform, we demonstrate an NOSP at a data rate approaching 40 gigabits per second with a record low pump power of one milliwatt. These findings pave the way for the development of fully chip-scale NOSP devices with pump light sources integrated together, potentially leading to a wide range of applications in optical communication networks and classical or quantum computation. The combination of PT symmetry and NOSP may also open up opportunities for amplification, detection, and sensing, where response speed and efficiency are equally important.Supplementary informationThe online version contains supplementary material available at 10.1186/s43593-024-00062-w.
| S-EPMC10995095 | biostudies-literature
Project description:Carbon sequestration in soils under agricultural use can contribute to climate change mitigation. Spatial-temporal soil organic carbon (SOC) monitoring requires more efficient data acquisition. This study aims to evaluate the potential of spectral on-the-go proximal measurements to serve these needs. The study was conducted as a long-term field experiment. SOC values ranged between 14 and 25 g kg-1 due to different fertilization treatments. Partial least squares regression models were built based on the spectral laboratory and field data collected with two spectrometers (site-specific and on-the-go). Correction of the field data based on the laboratory data was done by testing linear transformation, piecewise direct standardization, and external parameter orthogonalization (EPO). Different preprocessing methods were applied to extract the best possible information content from the sensor signal. The models were then thoroughly interpreted concerning spectral wavelength importance using regression coefficients and variable importance in projection scores. The detailed wavelength importance analysis disclosed the challenge of using soil spectroscopy for SOC monitoring. The use of different spectrometers under varying soil conditions revealed shifts in wavelength importance. Still, our findings on the use of on-the-go spectroscopy for spatial-temporal SOC monitoring are promising.
| S-EPMC10857020 | biostudies-literature
Project description:In estimating the frequency spectrum of real-world time series data, we must violate the assumption of infinite-length, orthogonal components in the Fourier basis. While it is widely known that care must be taken with discretely sampled data to avoid aliasing of high frequencies, less attention is given to the influence of low frequencies with period below the sampling time window. Here, we derive an analytic expression for the side-lobe attenuation of signal components in the frequency domain representation. This expression allows us to detail the influence of individual frequency components throughout the spectrum. The first consequence is that the presence of low-frequency components introduces a 1/f[Formula: see text] component across the power spectrum, with a scaling exponent of [Formula: see text]. This scaling artifact could be composed of diffuse low-frequency components, which can render it difficult to detect a priori. Further, treatment of the signal with standard digital signal processing techniques cannot easily remove this scaling component. While several theoretical models have been introduced to explain the ubiquitous 1/f[Formula: see text] scaling component in neuroscientific data, we conjecture here that some experimental observations could be the result of such data analysis procedures.
| S-EPMC5555851 | biostudies-other
Project description:Data on genome organization and output over time, or the 4D Nucleome (4DN), require synthesis for meaningful interpretation. Development of tools for the efficient integration of these data is needed, especially for the time dimension. We present the '4DNvestigator', a user-friendly network-based toolbox for the analysis of time series genome-wide genome structure (Hi-C) and gene expression (RNA-seq) data. Additionally, we provide methods to quantify network entropy, tensor entropy, and statistically significant changes in time series Hi-C data at different genomic scales.
| S-EPMC8049205 | biostudies-literature