Home » Hardware, Picture Gallery » Big Test Data software tool-chain as a perfect solution for data mining of measurement data

Big Test Data software tool-chain as a perfect solution for data mining of measurement data




 

When analyzing vast amount of data, one term inevitably becomes relevant: Data Mining. By applying statistical methods, countless data points are processed by algorithms for the recognition of patterns, coherences, and clusters amongst other things. The AMS offers a tool-chain that covers the complete workflow necessary for analyzing Big Data ? from data storage with MaDaM to parallel evaluation with jBEAM-Cluster and even including statistical analysis of these evaluation results (Test Data Mining).
The AMS BIG-TEST-DATA approach offers multiple technologies for distributed analysis from one provider. There are three core products used to obtain the maximum information from test and measurement data for engineers: 1. jBEAM: The analysis and visualization tool, Java based, useful for both server and desktop application, with 18 years of experience and continuous development. 2. MaDaM: The Measurement Data Management system (MDM), available now as version 2 with Elasticsearch indexing technology and modern web interfaces. 3. jBEAM-Cluster: Cluster management software for parallel and distributed analysis of a myriad of data files – an essential and time-saving process.
In the field of measurement data, the data volumes continue to increase exponentially. Without an organizational tool like MaDaM, no intelligent post-processing can be done. The subsequent data mining process also has its starting point in this central MDM system. The MaDaM importer selects information about each measured channel and analyzes it to extract statistical information besides the regular metadata. Data about the related engineer and test object as well as such statistical values are indexed and made available for later search queries.
The new version 2 of MaDaM was completely redesigned based on the 4 years of experience with version 1. Elasticsearch replaces the former Lucene technology and offers horizontal scalability without limits. A modern JavaScript based framework (ReactJS) lets MaDaM convince with fast page rendering and short response times.
Typically, the data mining process based on individual single measurement values starts with selecting a set of relevant tests. Relevant are all tests that can be used to answer the questions that are looked for. The number of such tests can be in the hundreds or in the hundreds of thousands. All of these files together with the definition of the analysis that should be performed is provided to the jBEAM-Cluster as a job. The files are analyzed in parallel and close to the storage location while taking into account each individual value of each channel. The statistical results of each file are aggregated and sent back to the originator of the job.
The final step is analyzing these results with data mining methodologies: jBEAM has a whole set of data mining algorithms, covering the approaches Clustering, Pattern Recognition, Prediction, and Reducing Dimension of Relation amongst others. After this data mining step, the calculated result can be visualized with dedicated graphs as a report. Each intermediate step can be controlled manually where the operator can optimize the boundary conditions of the data mining process in a highly interactive way.
These three steps focused on and optimized for measurement data are the essentials for the Test Data Mining solution of AMS.





Posted by on 27. June 2018. Filed under Hardware, Picture Gallery. You can follow any responses to this entry through the RSS 2.0. You can leave a response or trackback to this entry

Leave a Reply

Archive

© 2018 So-Co-IT. All Rights Reserved. Log in - Copyright by LayerMedia


Blogverzeichnis - Blog Verzeichnis bloggerei.de Blog Top Liste - by TopBlogs.de Bloggeramt.de