Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. In this study, we investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. Notably, experience gathered solely from benign tasks can still compromise safety in high-risk scenarios. Further analysis attributes this degradation to the execution-oriented nature of accumulated experience, which reinforces agents' tendency to act rather than refuse. In more realistic settings where agents encounter both benign and harmful tasks, refusal-related experience mitigates safety decline but induces over-refusal, revealing a fundamental safety-utility trade-off. Overall, our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.
@article{arxiv.2604.16968,
title = {On Safety Risks in Experience-Driven Self-Evolving Agents},
author = {Weixiang Zhao and Yichen Zhang and Yingshuo Wang and Yang Deng and Yanyan Zhao and Xuda Zhi and Yongbo Huang and HaoHe and Wanxiang Che and Bing Qin and Ting Liu},
journal= {arXiv preprint arXiv:2604.16968},
year = {2026}
}