English

Distributed Function Minimization in Apache Spark

Machine Learning 2019-09-18 v1 Optimization and Control Machine Learning
Authors: Andrea Schioppa
View on arXiv ↗ PDF ↗

Abstract

We report on an open-source implementation for distributed function minimization on top of Apache Spark by using gradient and quasi-Newton methods. We show-case it with an application to Optimal Transport and some scalability tests on classification and regression problems.

Cite

@article{arxiv.1909.07922,
  title  = {Distributed Function Minimization in Apache Spark},
  author = {Andrea Schioppa},
  journal= {arXiv preprint arXiv:1909.07922},
  year   = {2019}
}

Related papers

View all related →
Distributed, Parallel, and Cluster Computing · Computer Science
Understanding and Optimizing the Performance of Distributed Machine Learning Applications on Apache Spark
Celestine Dünner, Thomas Parnell, Kubilay Atasu, Manolis Sifalakis +1
2018-06-21
Databases · Computer Science
On the Evaluation of RDF Distribution Algorithms Implemented over Apache Spark
Olivier Curé, Hubert Naacke, Mohamed-Amine Baazizi, Bernd Amann
2015-07-10
Programming Languages · Computer Science
Distributed Programming via Safe Closure Passing
Philipp Haller, Heather Miller
2016-02-12
Machine Learning · Computer Science
Modeling Scalability of Distributed Machine Learning
Alexander Ulanov, Andrey Simanovsky, Manish Marwah
2017-03-28
Methodology · Statistics
Shared active subspace for multivariate vector-valued functions
Khadija Musayeva, Mickael Binois
2024-01-08
Optimization and Control · Mathematics
Distributed accelerated proximal conjugate gradient methods for multi-agent constrained optimization problems
Anteneh Getachew Gebrie
2024-06-21
Distributed, Parallel, and Cluster Computing · Computer Science
Technical Report: On the Usability of Hadoop MapReduce, Apache Spark & Apache Flink for Data Science
Bilal Akil, Ying Zhou, Uwe Röhm
2018-03-30
Optimization and Control · Mathematics
Distributed Adaptive Gradient Algorithm with Gradient Tracking for Stochastic Non-Convex Optimization
Dongyu Han, Kun Liu, Yeming Lin, Yuanqing Xia
2024-04-01
Machine Learning · Statistics
Optimization for Large-Scale Machine Learning with Distributed Features and Observations
Alexandros Nathan, Diego Klabjan
2017-04-18
Machine Learning · Statistics
A Stochastic Large-scale Machine Learning Algorithm for Distributed Features and Observations
Biyi Fang, Diego Klabjan
2019-12-10
Optimization and Control · Mathematics
Distributed optimization on directed graphs based on inexact ADMM with partial participation
Dingran Yi, Nikolaos M. Freris
2023-04-04
Optimization and Control · Mathematics
Distributed Derivative-free Learning Method for Stochastic Optimization over a Network with Sparse Activity
Wenjie Li, Mohamad Assaad, Shiqi Zheng
2021-04-20
Machine Learning · Computer Science
On Scalable and Efficient Computation of Large Scale Optimal Transport
Yujia Xie, Minshuo Chen, Haoming Jiang, Tuo Zhao +1
2019-06-26
Optimization and Control · Mathematics
Decentralized Min-Max Optimization with Gradient Tracking
Runze You, Kun Huang, Shi Pu
2025-05-19
Optimization and Control · Mathematics
Distributed Stochastic Nonsmooth Nonconvex Optimization
Vyacheslav Kungurtsev
2019-11-05
Distributed, Parallel, and Cluster Computing · Computer Science
A duality-based approach for distributed min-max optimization with application to demand side management
Ivano Notarnicola, Mauro Franceschelli, Giuseppe Notarstefano
2017-03-27
Machine Learning · Computer Science
Distributed stochastic optimization with gradient tracking over strongly-connected networks
Ran Xin, Anit Kumar Sahu, Usman A. Khan, Soummya Kar
2019-04-11
Optimization and Control · Mathematics
A Distributed Newton Method for Network Utility Maximization
Ermin Wei, Asuman Ozdaglar, Ali Jadbabaie
2015-03-17
Distributed, Parallel, and Cluster Computing · Computer Science
Solving All-Pairs Shortest-Paths Problem in Large Graphs Using Apache Spark
Frank Schoeneman, Jaroslaw Zola
2019-08-08
Distributed, Parallel, and Cluster Computing · Computer Science
Approximate Distributed Joins in Apache Spark
Do Le Quoc, Istemi Ekin Akkus, Pramod Bhatotia, Spyros Blanas +3
2018-05-16
Optimization and Control · Mathematics
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich +1
2019-04-11
Optimization and Control · Mathematics
Distributed Newton Optimization with Maximized Convergence Rate
Damián Marelli, Yong Xu, Minyue Fu, Zenghong Huang
2022-06-07
Computation · Statistics
Optimizing and accelerating space-time Ripley's K function based on Apache Spark for distributed spatiotemporal point pattern analysis
Yuan Wang, Zhipeng Gui, Huayi Wu, Dehua Peng +2
2019-12-11
Machine Learning · Computer Science
An introduction to decentralized stochastic optimization with gradient tracking
Ran Xin, Soummya Kar, Usman A. Khan
2019-11-14
Optimization and Control · Mathematics
A Fast Gradient and Function Sampling Method for Finite Max-Functions
Elias S. Helou, Sandra A. Santos, Lucas E. A. Simões
2019-04-03
R2 v1 2026-06-23T11:18:10.046Z