English

Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems

Artificial Intelligence 2026-05-08 v1

Abstract

LLM-based multi-agent systems are increasingly deployed for payment workflows, yet prevailing metrics, Task Success Rate (TSR) and Agent Handoff F1-Score (HF1), capture only final outcomes or unordered routing decisions. We introduce the Agentic Success Rate (ASR), a trajectory-fidelity metric that compares observed and expected agent execution sequences at the transition level, decomposing performance into Transition Recall and Transition Precision. Applied to the Hierarchical Multi-Agent System for Payments (HMASP) across 18 LLMs and 90,000 task instances, ASR reveals that 10 of 18 models systematically skip a confirmation checkpoint during payment checkout, a deviation invisible to both TSR and HF1, while 8 models enforce the checkpoint perfectly. Notably, GPT-4.1 exhibits hidden workflow shortcuts despite achieving perfect TSR and HF1, while GPT-5.2 achieves perfect ASR. Prompt refinements and deterministic routing guards guided by ASR diagnostics yield substantial TSR improvements, with gains up to +93.8 percentage points for previously struggling models, demonstrating that trajectory-level evaluation is essential in regulated domains.

Keywords

Cite

@article{arxiv.2605.06457,
  title  = {Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems},
  author = {Donghao Huang and Joon Kiat Chua and Zhaoxia Wang},
  journal= {arXiv preprint arXiv:2605.06457},
  year   = {2026}
}

Comments

6 pages, 2 tables. Accept at AI and Data Science for Digital Finance (AIDS4DF) Workshop, PAKDD 2026

R2 v1 2026-07-01T12:55:23.793Z