English

Adaptive Execution Scheduler for DataDios SmartDiff

Distributed, Parallel, and Cluster Computing 2025-10-10 v1 Machine Learning

Abstract

We present an adaptive scheduler for a single differencing engine (SmartDiff) with two execution modes: (i) in-memory threads and (ii) Dask based parallelism. The scheduler continuously tunes batch size and worker/thread count within fixed CPU and memory budgets to minimize p95 latency. A lightweight preflight profiler estimates bytes/row and I/O rate; an online cost/memory model prunes unsafe actions; and a guarded hill-climb policy favors lower latency with backpressure and straggler mitigation. Backend selection is gated by a conservative working-set estimate so that in-memory execution is chosen when safe, otherwise Dask is used. Across synthetic and public tabular benchmarks, the scheduler reduces p95 latency by 23 to 28 percent versus a tuned warm-up heuristic (and by 35 to 40 percent versus fixed grid baselines), while lowering peak memory by 16 to 22 percent (25 to 32 percent vs. fixed) with zero OOMs and comparable throughput.

Keywords

Cite

@article{arxiv.2510.07811,
  title  = {Adaptive Execution Scheduler for DataDios SmartDiff},
  author = {Aryan Poduri},
  journal= {arXiv preprint arXiv:2510.07811},
  year   = {2025}
}

Comments

4 pages, 1 figure

R2 v1 2026-07-01T06:25:48.276Z