A widely used standard for the persistent storage of measurement data is ASAM ODS. Automotive manufacturers and suppliers use it primarily to set up comprehensive data management solutions for different measurement systems and data formats. In this way, they make test data usable in the long-term, regardless of the tools of individual vendors. Generally in such solutions, the search for certain measurements bases on descriptive information, the so-called metadata.
To gain more information from the existing measurement channels, Peak Solution has developed the Peak Advanced Search.
The solution allows the user to search for information directly in the channel data of ASAM ODS (= external components). After selecting a set of measurement channels, he can execute arbitrarily complex SQL queries on it. The solution returns the query result to the user as a list. He can either cache this list or export it as a csv file, e.g. in order to apply algorithms and statistical methods for detecting patterns, relationships, and clusters.
The implementation of Peak Advanced Search bases on Apache Spark. This freely available framework makes it possible to perform complex data queries and analyzes on large amounts of data with high speed. In order to access ASAM ODS Peak Solution has developed a special plug-in. This provides the data structure of the measurement channels in Spark as so-called RDDs (Resilient Distributed Datasets). Out of this the application then creates temporary tables (= data frames) on which the mentioned SQL queries can be executed.
The user can use Apache Zeppelin for the interactive analysis of the measurement channels. The great strength of this web-based “notebook” are explorative evaluations (= data mining). This allows the user first to gain an overview of the data and then approach his analysis goal gradually. He can easily visualize and share intermediate results. In order to do this, the user only has to release his “notebook” as a web page. Authorized persons can then view and continue to use it.
In addition, Peak Solution has ensured that less experienced users can use the Peak Advanced Search out of the Peak Test Data Manager as well. For this purpose, the openMDM-based solution has been expanded by special dialog components and communication interfaces.
An important advantage of the Peak Advanced Search is its scalability: neither Spark nor Zeppelin require a Hadoop Eco-System. As a result, companies can leverage the solution within their existing ODS infrastructure without extensive IT investments before gradually building it into a powerful big data cluster for managing and analyzing measurement data. According to the motto: Start small, think tall.