Efficient Off-Policy Reinforcement Learning via Brain-Inspired Computing
Abstract
Reinforcement Learning (RL) has opened up new opportunities to enhance existing smart systems that generally include a complex decision-making process. However, modern RL algorithms, e.g., Deep Q-Networks (DQN), are based on deep neural networks, resulting in high computational costs. In this paper, we propose QHD, an off-policy value-based Hyperdimensional Reinforcement Learning, that mimics brain properties toward robust and real-time learning. QHD relies on a lightweight brain-inspired model to learn an optimal policy in an unknown environment. On both desktop and power-limited embedded platforms, QHD achieves significantly better overall efficiency than DQN while providing higher or comparable rewards. QHD is also suitable for highly-efficient reinforcement learning with great potential for online and real-time learning. Our solution supports a small experience replay batch size that provides 12.3 times speedup compared to DQN while ensuring minimal quality loss. Our evaluation shows QHD capability for real-time learning, providing 34.6 times speedup and significantly better quality of learning than DQN.
Cite
@article{arxiv.2205.06978,
title = {Efficient Off-Policy Reinforcement Learning via Brain-Inspired Computing},
author = {Yang Ni and Danny Abraham and Mariam Issa and Yeseong Kim and Pietro Mercati and Mohsen Imani},
journal= {arXiv preprint arXiv:2205.06978},
year = {2023}
}
Comments
In Proceedings of the Great Lakes Symposium on VLSI 2023(GLSVLSI 2023)