Integration of Skyline Queries into Spark SQL
Abstract
Skyline queries are frequently used in data analytics and multi-criteria decision support applications to filter relevant information from big amounts of data. Apache Spark is a popular framework for processing big, distributed data. The framework even provides a convenient SQL-like interface via the Spark SQL module. However, skyline queries are not natively supported and require tedious rewriting to fit the SQL standard or Spark's SQL-like language. The goal of our work is to fill this gap. We thus provide a full-fledged integration of the skyline operator into Spark SQL. This allows for a simple and easy to use syntax to input skyline queries. Moreover, our empirical results show that this integrated solution of skyline queries by far outperforms a solution based on rewriting into standard SQL.
Keywords
Cite
@article{arxiv.2210.03718,
title = {Integration of Skyline Queries into Spark SQL},
author = {Lukas Grasmann and Reinhard Pichler and Alexander Selzer},
journal= {arXiv preprint arXiv:2210.03718},
year = {2022}
}