Data storage trying to keep up with a speeding car

Development and testing of autonomous vehicle technology is data intensive, requiring solutions for the management and storage and of rapidly expanding volumes of data.

The primary reason for this explosion is the collection of video and LiDAR data, which has to be high-resolution. Naturally, the more sensors one has on the AV, the more data the AV would generate. An AV sensor suite may comprise of eight cameras, two LiDAR sensors and two radars sensors or it may comprise of just one LiDAR and a couple of cameras.

Depending on the operating conditions, for example the operational design domain (ODD), of the automated driving system (ADS), the manufacturers’ choice of sensor suite may differ substantially. For example, the resolution of data required for a low-speed ADS application in a business park (constrained ODD) may be different to a high-speed application on highways.

“Vehicles are collecting data 24/7 as they create a 3D world via LiDAR and video and there is no easy way to compress data,” explained Ken Obuszewski, global general manager of NetApp’s automotive vertical. “Traditional methods of data reduction don’t work in this context.” He said when it comes to managing this data, it is critical to have rich metadata about the data you have captured in order to optimize the processing of the data, while data tiering and archiving to the cloud are necessary to store and retain massive amounts of data.

Data management methods such as tiering and archiving allow for proper insight into what data gets stored, and what data gets discarded. For storing the data, Obuszewski said that because these huge volumes of data are generated in multiple locations, the challenge is how to provide and access that data where and when you need it. In short – flexibility and locality matters.

To read the complete article, visit TU-Automotive.