English

Exoshuffle-CloudSort

Distributed, Parallel, and Cluster Computing 2023-01-11 v1 Operating Systems
Authors: Frank Sifei Luan , Stephanie Wang , Samyukta Yagati , Sean Kim , Kenneth Lien , Isaac Ong , Tony Hong , SangBin Cho , Eric Liang , Ion Stoica
View on arXiv ↗ PDF ↗

Abstract

We present Exoshuffle-CloudSort, a sorting application running on top of Ray using the Exoshuffle architecture. Exoshuffle-CloudSort runs on Amazon EC2, with input and output data stored on Amazon S3. Using 40 i4i.4xlarge workers, Exoshuffle-CloudSort completes the 100 TB CloudSort Benchmark (Indy category) in 5378 seconds, with an average total cost of $97.

Cite

@article{arxiv.2301.03734,
  title  = {Exoshuffle-CloudSort},
  author = {Frank Sifei Luan and Stephanie Wang and Samyukta Yagati and Sean Kim and Kenneth Lien and Isaac Ong and Tony Hong and SangBin Cho and Eric Liang and Ion Stoica},
  journal= {arXiv preprint arXiv:2301.03734},
  year   = {2023}
}

Related papers

View all related →
Distributed, Parallel, and Cluster Computing · Computer Science
Exoshuffle: An Extensible Shuffle Architecture
Frank Sifei Luan, Stephanie Wang, Samyukta Yagati, Sean Kim +6
2023-08-21
Data Structures and Algorithms · Computer Science
Sorting Data on Ultra-Large Scale with RADULS. New Incarnation of Radix Sort
Marek Kokot, Sebastian Deorowicz, Agnieszka Debudaj-Grabysz
2016-12-09
Distributed, Parallel, and Cluster Computing · Computer Science
Coded TeraSort
Songze Li, Sucha Supittayapornpong, Mohammad Ali Maddah-Ali, A. Salman Avestimehr
2017-02-17
Instrumentation and Methods for Astrophysics · Physics
The Application of Cloud Computing to Astronomy: A Study of Cost and Performance
G. Bruce Berriman, Ewa Deelman, Gideon Juve, Moira Regelson +1
2015-03-17
Distributed, Parallel, and Cluster Computing · Computer Science
From Sand to Flour: The Next Leap in Granular Computing with NanoSort
Theo Jepsen, Stephen Ibanez, Gregory Valiant, Nick McKeown
2022-04-28
Distributed, Parallel, and Cluster Computing · Computer Science
Performance Measurements of Supercomputing and Cloud Storage Solutions
Michael Jones, Jeremy Kepner, William Arcand, David Bestor +9
2018-03-06
Distributed, Parallel, and Cluster Computing · Computer Science
FuxiShuffle: An Adaptive and Resilient Shuffle Service for Distributed Data Processing on Alibaba Cloud
Yuhao Lin, Zhipeng Tang, Jiayan Tong, Junqing Xiao +10
2026-02-27
Databases · Computer Science
A Memory Bandwidth-Efficient Hybrid Radix Sort on GPUs
Elias Stehle, Hans-Arno Jacobsen
2017-05-22
Databases · Computer Science
WiscSort: External Sorting For Byte-Addressable Storage
Vinay Banakar, Kan Wu, Yuvraj Patel, Kimberly Keeton +2
2023-07-14
Distributed, Parallel, and Cluster Computing · Computer Science
SEARS: Space Efficient And Reliable Storage System in the Cloud
Ying Li, Katherine Guo, Xin Wang, Emina Soljanin +1
2016-11-18
Data Structures and Algorithms · Computer Science
zSort: Stable Distribution Sort using Z-Score Partitioning
Hriday Jain, Ketan Sabale, Aditya Shastri, Hiren Kumar Thakkar +1
2026-05-15
Databases · Computer Science
Performance / Price Sort
Jim Gray, Joshua Coates, Chris Nyberg
2007-05-23
Networking and Internet Architecture · Computer Science
FAST CLOUD: Pushing the Envelope on Delay Performance of Cloud Storage with Coding
Guanfeng Liang, Ulas C. Kozat
2013-11-04
Data Structures and Algorithms · Computer Science
TSSort: Probabilistic Noise Resistant Sorting
Jörn Hees, Benjamin Adrian, Ralf Biedert, Thomas Roth-Berghofer +1
2016-06-17
Databases · Computer Science
Parallel External Sorting of ASCII Records Using Learned Models
Ani Kristo, Tim Kraska
2023-05-11
Distributed, Parallel, and Cluster Computing · Computer Science
Engineering In-place (Shared-memory) Sorting Algorithms
Michael Axtmann, Sascha Witt, Daniel Ferizovic, Peter Sanders
2021-02-05
Distributed, Parallel, and Cluster Computing · Computer Science
Location, Location, Location: Data-Intensive Distributed Computing in the Cloud
Michael Luckeneder, Adam Barker
2014-02-04
Data Structures and Algorithms · Computer Science
Faster Radix Sort via Virtual Memory and Write-Combining
Jan Wassenberg, Peter Sanders
2010-09-07
Distributed, Parallel, and Cluster Computing · Computer Science
FractalSortCPU: Bandwidth-Efficient Compressed Radix Sort on CPU
Michael Dang'ana
2026-05-13
Distributed, Parallel, and Cluster Computing · Computer Science
EvoSort: A Genetic-Algorithm-Based Adaptive Parallel Sorting Framework for Large-Scale High Performance Computing
Shashank Raj, Kalyanmoy Deb
2026-01-05
Data Structures and Algorithms · Computer Science
DialSort: Non-Comparative Integer Sorting via the Self-Indexing Principle: Architecture, Implementation, and Substrate-Aware Analysis
Alexander Narvaez
2026-05-19
Software Engineering · Computer Science
RIOT: a Stochastic-based Method for Workflow Scheduling in the Cloud
Jianfeng Chen, Tim Menzies
2018-04-24
Distributed, Parallel, and Cluster Computing · Computer Science
Flora: Efficient Cloud Resource Selection for Big Data Processing via Job Classification
Jonathan Will, Lauritz Thamsen, Jonathan Bader, Odej Kao
2025-03-03
Distributed, Parallel, and Cluster Computing · Computer Science
InfiniStore: Elastic Serverless Cloud Storage
Jingyuan Zhang, Ao Wang, Xiaolong Ma, Benjamin Carver +7
2023-03-20
Instrumentation and Methods for Astrophysics · Physics
Scientific Workflow Applications on Amazon EC2
Gideon Juve, Ewa Deelman, Karan Vahi, Gaurang Mehta +3
2016-11-15
R2 v1 2026-06-28T08:08:09.923Z