ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Abstract
Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.
Keywords
Cite
@article{arxiv.2604.23781,
title = {ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents},
author = {Fanqing Meng and Lingxiao Du and Zijian Wu and Guanzheng Chen and Xiangyan Liu and Jiaqi Liao and Chonghe Jiang and Zhenglin Wan and Jiawei Gu and Pengfei Zhou and Rui Huang and Ziqi Zhao and Shengyuan Ding and Ailing Yu and Bo Peng and Bowei Xia and Hao Sun and Haotian Liang and Ji Xie and Jiajun Chen and Jiajun Song and Liu Yang and Ming Xu and Qionglin Qiu and Runhao Fu and Shengfang Zhai and Shijian Wang and Tengfei Ma and Tianyi Wu and Weiyang Jin and Yan Wang and Yang Dai and Yao Lai and Youwei Shu and Yue Liu and Yunzhuo Hao and Yuwei Niu and Jinkai Huang and Jiayuan Zhuo and Zhennan Shen and Linyu Wu and Hannah Yao and Charles Chen and Cihang Xie and Yuyin Zhou and Jiaheng Zhang and Zeyu Zheng and Mengkang Hu and Michael Qizhe Shieh},
journal= {arXiv preprint arXiv:2604.23781},
year = {2026}
}
Comments
github repo: https://github.com/evolvent-ai/ClawMark