How to Evaluate Distributed Coordination Systems? -- A Survey and Analysis
Abstract
Coordination services and protocols are critical components of distributed systems and are essential for providing consistency, fault tolerance, and scalability. However, due to the lack of standard benchmarking and evaluation tools for distributed coordination services, coordination service developers/researchers either use a NoSQL standard benchmark and omit evaluating consistency, distribution, and fault tolerance; or create their own ad-hoc microbenchmarks and skip comparability with other services. In this study, we analyze and compare the evaluation mechanisms for known and widely used consensus algorithms, distributed coordination services, and distributed applications built on top of these services. We identify the most important requirements of distributed coordination service benchmarking, such as the metrics and parameters for the evaluation of the performance, scalability, availability, and consistency of these systems. Finally, we discuss why the existing benchmarks fail to address the complex requirements of distributed coordination system evaluation.
Cite
@article{arxiv.2403.09445,
title = {How to Evaluate Distributed Coordination Systems? -- A Survey and Analysis},
author = {Bekir Turkkan and Elvis Rodrigues and Tevfik Kosar and Aleksey Charapko and Ailidani Ailijiang and Murat Demirbas},
journal= {arXiv preprint arXiv:2403.09445},
year = {2025}
}
Comments
The paper has been revised and extended to include (1) additional distributed coordination algorithms and (2) additional benchmarking tools in the survey. Also, the selection criteria of the algorithms, services, and tools was specified