Efficient Sampled Softmax for Tensorflow
Machine Learning
2020-04-14 v1
Authors:
Maciej Skorski
Abstract
This short paper discusses an efficient implementation of \emph{sampled softmax loss} for Tensorflow. The speedup over the default implementation is achieved due to simplification of the graph for the forward and backward passes.
Cite
@article{arxiv.2004.05244,
title = {Efficient Sampled Softmax for Tensorflow},
author = {Maciej Skorski},
journal= {arXiv preprint arXiv:2004.05244},
year = {2020}
}
Related papers
View all related →
Mathematical Software · Computer Science
Softmax Optimizations for Intel Xeon Processor-based Platforms
Jacek Czaja, Michal Gallus, Tomasz Patejko, Jian Tang
2019-05-28
Numerical Analysis · Mathematics
Accurate Computation of the Log-Sum-Exp and Softmax Functions
Pierre Blanchard, Desmond J. Higham, Nicholas J. Higham
2019-09-10
Computation and Language · Computer Science
Speeding Up Entmax
Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé, Zhenisbek Assylbekov
2022-05-20
Machine Learning · Computer Science
Efficient Image Representation Learning with Federated Sampled Softmax
Sagar M. Waghmare, Hang Qi, Huizhong Chen, Mikhail Sirotenko +1
2022-03-10
Machine Learning · Computer Science
Sampled Softmax with Random Fourier Features
Ankit Singh Rawat, Jiecao Chen, Felix Yu, Ananda Theertha Suresh +1
2020-01-01
Hardware Architecture · Computer Science
Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany +1
2021-03-18
Machine Learning · Computer Science
Sparse-softmax: A Simpler and Faster Alternative Softmax Transformation
Shaoshi Sun, Zhenyuan Zhang, BoCheng Huang, Pengbin Lei +3
2021-12-24
Audio and Speech Processing · Electrical Eng. & Systems
Memory-Efficient Training of RNN-Transducer with Sampled Softmax
Jaesong Lee, Lukas Lee, Shinji Watanabe
2022-04-01
Machine Learning · Computer Science
A Tale of Two Efficient and Informative Negative Sampling Distributions
Shabnam Daghaghi, Tharun Medini, Nicholas Meisburger, Beidi Chen +2
2021-07-30
Hardware Architecture · Computer Science
Flex-TPU: A Flexible TPU with Runtime Reconfigurable Dataflow Architecture
Mohammed Elbtity, Peyton Chandarana, Ramtin Zand
2024-07-12
Machine Learning · Computer Science
Data-Efficient Training by Evolved Sampling
Ziheng Cheng, Zhong Li, Jiang Bian
2025-09-30
Performance · Computer Science
Online normalizer calculation for softmax
Maxim Milakov, Natalia Gimelshein
2018-07-31
Machine Learning · Computer Science
XES Tensorflow - Process Prediction using the Tensorflow Deep-Learning Framework
Joerg Evermann, Jana-Rebecca Rehse, Peter Fettke
2017-05-04
Hardware Architecture · Computer Science
Transparent FPGA Acceleration with TensorFlow
Simon Pfenning, Philipp Holzinger, Marc Reichenbach
2021-02-12
Distributed, Parallel, and Cluster Computing · Computer Science
TensorFlow Doing HPC
Steven W. D. Chien, Stefano Markidis, Vyacheslav Olshevsky, Yaroslav Bulatov +2
2020-03-03
Computation and Language · Computer Science
Efficient softmax approximation for GPUs
Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier +1
2017-06-20
Machine Learning · Computer Science
Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications
Jin Chen, Jin Zhang, Xu huang, Yi Yang +2
2025-01-16
Machine Learning · Computer Science
Swift for TensorFlow: A portable, flexible platform for deep learning
Brennan Saeta, Denys Shabalin, Marc Rasi, Brad Larson +9
2021-03-01
Quantum Physics · Physics
TensorTrace: an application to contract tensor networks
Glen Evenbly
2019-11-07
Distributed, Parallel, and Cluster Computing · Computer Science
TensorFlow: A system for large-scale machine learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen +18
2016-06-01
Artificial Intelligence · Computer Science
Equality Saturation for Tensor Graph Superoptimization
Yichen Yang, Phitchaya Mangpo Phothilimtha, Yisu Remy Wang, Max Willsey +2
2021-03-18
Computation and Language · Computer Science
Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs
Sachin Kumar, Yulia Tsvetkov
2019-03-25
Distributed, Parallel, and Cluster Computing · Computer Science
Taskflow: A Lightweight Parallel and Heterogeneous Task Graph Computing System
Tsung-Wei Huang, Dian-Lun Lin, Chun-Xun Lin, Yibo Lin
2021-09-08
Optimization and Control · Mathematics
Accelerated Proximal Envelopes: Application to the Coordinate Descent Method
Dmitry Pasechnyuk, Anton Anikin, Vladislav Matyukhin
2021-01-14
Hardware Architecture · Computer Science
A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks
Anthony Leiva-Valverde, Fabricio Elizondo-Fernández, Luis G. León-Vega, Cristina Meinhardt +1
2026-04-09