Smart meter data processing: a showcase for simple and efficient textual processing
Abstract
The increase in the production and collection of data from devices is an ongoing trend due to the roll-out of more cyber-physical applications. Smart meters, because of their importance in power grids, are a class of such devices whose produced data requires meticulous processing. In this paper, we use Unicage, a data processing system based on classic Unix shell scripting, that delivers excellent performance in a simple package. We use this methodology to process smart meter data in XML format, subjected to the constraints posed by a real use case. We develop a solution that parses, validates and performs a simple aggregation of 27 million XML files in less than 10 minutes. We present a study of the solution as well as the benefits of its adoption.
Cite
@article{arxiv.2212.13656,
title = {Smart meter data processing: a showcase for simple and efficient textual processing},
author = {Miguel Ferreira and André Neves and Rodrigo Gorjão and Carlos Cruz and Miguel L. Pardal},
journal= {arXiv preprint arXiv:2212.13656},
year = {2022}
}
Comments
11 pages, 5 figures, 1 table, 9 listings. Accepted after review for the 1st Workshop on High-Performance and Reliable Big Data (HPBD 2021), which was held virtually on September 20th 2021, and was co-located with the 40th International Symposium on Reliable Distributed Systems (SRDS 2021)