Automotive companies generate large amounts of test data from various sources such as test benches, simulations, driving and field tests. This data volume is growing exponentially day-by-day. Because traditional solutions reach their performance limits, the industry is dealing for some time with the question of how Big Data technologies can be beneficially used in the field of test data management and analysis. Frequently the companies desire to enhance established standards like ASAM ODS gradually.
The development of index-based search methods as well as new query and analysis techniques for ODS data are two concrete project examples.
Index-based search methods:
In the context of several customer projects Peak Solution has developed a full-text search for openMDM 4 based on ElasticSearch. Part of the solution is a “crawler”, which continuously generate a file-based index from the connected ODS data. The index contains all the metadata stored in ODS data objects (for example, project, test, test step and measurement). The access to the data takes place via the Peak ODS Server. Additionally, the Peak ODS Notification Manager ensures that the data in the index are always up to date. In search queries, the ElasticSearch server accesses the index. The search results are then returned via the REST interface of the server. For entering search queries and displaying search results, Peak Solution provides a special openMDM 4 FullTextSearch client. With this solution, it is possible to implement a cross-site search based on openMDM 4. However, the local deployment of remote mass data (for example, measurement files) via weak network connections is thus not yet solved. For this, other methods must be implemented at the application level (for example, triggering of an automated file transfers based on ATFx)
Query and analysis techniques:
Apache Spark is an open source framework, with which large amounts of data in a Hadoop cluster can be processed extremely efficient. To make this possible with existing ODS data stocks and without extensive data conversions, Peak Solution has developed the prototype of a File Input Format for ASAM MDF (= Mass Data Files). By that, the ODS data are presented in a data structure, which is suitable for Spark operations. In this way, Big Data Scientists can analyze existing ODS data – even without special ODS knowledge (!) – using Apache Spark Tools.
More BigODS projects are in preparation. We will report on it at a given time.