AutoReP: Automatic ReLU Replacement for Fast Private Network Inference
Abstract
The growth of the Machine-Learning-As-A-Service (MLaaS) market has highlighted clients' data privacy and security issues. Private inference (PI) techniques using cryptographic primitives offer a solution but often have high computation and communication costs, particularly with non-linear operators like ReLU. Many attempts to reduce ReLU operations exist, but they may need heuristic threshold selection or cause substantial accuracy loss. This work introduces AutoReP, a gradient-based approach to lessen non-linear operators and alleviate these issues. It automates the selection of ReLU and polynomial functions to speed up PI applications and introduces distribution-aware polynomial approximation (DaPa) to maintain model expressivity while accurately approximating ReLUs. Our experimental results demonstrate significant accuracy improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%, 12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget, Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever, AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55% accuracy with 176.1 times ReLU budget reduction.
Keywords
Cite
@article{arxiv.2308.10134,
title = {AutoReP: Automatic ReLU Replacement for Fast Private Network Inference},
author = {Hongwu Peng and Shaoyi Huang and Tong Zhou and Yukui Luo and Chenghong Wang and Zigeng Wang and Jiahui Zhao and Xi Xie and Ang Li and Tony Geng and Kaleel Mahmood and Wujie Wen and Xiaolin Xu and Caiwen Ding},
journal= {arXiv preprint arXiv:2308.10134},
year = {2023}
}
Comments
ICCV 2023 accepeted publication