Multi-GPU Training of ConvNets
Machine Learning
2014-02-20 v4 Neural and Evolutionary Computing
Abstract
In this work we evaluate different approaches to parallelize computation of convolutional neural networks across several GPUs.
Keywords
Cite
@article{arxiv.1312.5853,
title = {Multi-GPU Training of ConvNets},
author = {Omry Yadan and Keith Adams and Yaniv Taigman and Marc'Aurelio Ranzato},
journal= {arXiv preprint arXiv:1312.5853},
year = {2014}
}
Comments
Machine Learning, Deep Learning, Convolutional Networks, Computer Vision, GPU, CUDA
Related papers
View all related →
Neural and Evolutionary Computing · Computer Science
One weird trick for parallelizing convolutional neural networks
Alex Krizhevsky
2014-04-29
Machine Learning · Computer Science
Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
Zhihao Jia, Sina Lin, Charles R. Qi, Alex Aiken
2018-06-12
Machine Learning · Computer Science
Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid
Andrew C. Kirby, Siddharth Samsi, Michael Jones, Albert Reuther +2
2020-09-01
Distributed, Parallel, and Cluster Computing · Computer Science
Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs
Behnam Pourghassemi, Chenghao Zhang, Joo Hwan Lee, Aparna Chandramowlishwaran
2020-05-29
Numerical Analysis · Mathematics
Parareal Neural Networks Emulating a Parallel-in-time Algorithm
Chang-Ock Lee, Youngkyu Lee, Jongho Park
2024-07-08
Machine Learning · Computer Science
Optimizing Performance of Recurrent Neural Networks on GPUs
Jeremy Appleyard, Tomas Kocisky, Phil Blunsom
2016-04-08
Distributed, Parallel, and Cluster Computing · Computer Science
A Multi-signal Variant for the GPU-based Parallelization of Growing Self-Organizing Networks
Giacomo Parigi, Angelo Stramieri, Danilo Pau, Marco Piastra
2015-03-31
Distributed, Parallel, and Cluster Computing · Computer Science
Characterizing and Understanding Distributed GNN Training on GPUs
Haiyang Lin, Mingyu Yan, Xiaocheng Yang, Mo Zou +3
2022-04-19
Distributed, Parallel, and Cluster Computing · Computer Science
Large Scale Artificial Neural Network Training Using Multi-GPUs
Linnan Wang, Wei Wu, Jianxiong Xiao, Yang Yi
2015-11-16
Computer Vision and Pattern Recognition · Computer Science
GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
Thomas Paine, Hailin Jin, Jianchao Yang, Zhe Lin +1
2013-12-24
Neural and Evolutionary Computing · Computer Science
Single stream parallelization of generalized LSTM-like RNNs on a GPU
Kyuyeon Hwang, Wonyong Sung
2015-11-25
Artificial Intelligence · Computer Science
Experimenting with Constraint Programming on GPU
Fabio Tardivo
2019-09-23
Numerical Analysis · Computer Science
Parallelizing Over Artificial Neural Network Training Runs with Multigrid
Jacob B. Schroder
2017-10-03
Distributed, Parallel, and Cluster Computing · Computer Science
Distributed learning of CNNs on heterogeneous CPU/GPU architectures
Jose Marques, Gabriel Falcao, Luís A. Alexandre
2017-12-08
Machine Learning · Computer Science
Gear Training: A new way to implement high-performance model-parallel training
Hao Dong, Shuai Li, Dongchang Xu, Yi Ren +1
2018-06-12
Neural and Evolutionary Computing · Computer Science
Parallel and in-process compilation of individuals for genetic programming on GPU
Hakan Ayral, Songül Albayrak
2017-05-23
Machine Learning · Computer Science
Exploring the power of GPU's for training Polyglot language models
Vivek Kulkarni, Rami Al-Rfou', Bryan Perozzi, Steven Skiena
2014-04-16
Machine Learning · Computer Science
Parallel training of linear models without compromising convergence
Nikolas Ioannou, Celestine Dünner, Kornilios Kourtis, Thomas Parnell
2018-12-20
Distributed, Parallel, and Cluster Computing · Computer Science
Hybrid Data-Model Parallel Training for Sequence-to-Sequence Recurrent Neural Network Machine Translation
Junya Ono, Masao Utiyama, Eiichiro Sumita
2019-09-10
Distributed, Parallel, and Cluster Computing · Computer Science
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
Chi-Chung Chen, Chia-Lin Yang, Hsiang-Yun Cheng
2019-10-29
Machine Learning · Computer Science
Multi-task parallelism for robust pre-training of graph foundation models on multi-source, multi-fidelity atomistic modeling data
Massimiliano Lupo Pasini, Jong Youl Choi, Pei Zhang, Kshitij Mehta +5
2025-06-30
Machine Learning · Computer Science
Accelerating GAN training using highly parallel hardware on public cloud
Renato Cardoso, Dejan Golubovic, Ignacio Peluaga Lozada, Ricardo Rocha +2
2021-11-09
Machine Learning · Computer Science
Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures
Felipe Costa Farias, Teresa Bernarda Ludermir, Carmelo Jose Albanez Bastos-Filho
2022-06-20
Machine Learning · Computer Science
Experiments on Parallel Training of Deep Neural Network using Model Averaging
Hang Su, Haoyu Chen
2018-07-03
Distributed, Parallel, and Cluster Computing · Computer Science
Deep Learning and Machine Learning with GPGPU and CUDA: Unlocking the Power of Parallel Computing
Ming Li, Ziqian Bi, Tianyang Wang, Yizhu Wen +13
2025-11-20