Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis
Abstract
Developing and deploying AI applications in organizations is challenging when human decision authority and oversight are underspecified across the system lifecycle. Although Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged, operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented. We report a multi-source qualitative study: a retrospective diary study of a customer-support chatbot and semi-structured interviews with eight AI experts from academia and industry. Through five-cycle thematic analysis of 1,435 codewords, we derive four themes: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination. These themes provide empirical inputs for subsequent HITL framework design and validation.
Cite
@article{arxiv.2603.05510,
title = {Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis},
author = {Parm Suksakul and Nathan Kittichaikoonkij and Nakhin Polthai and Aung Pyae},
journal= {arXiv preprint arXiv:2603.05510},
year = {2026}
}
Comments
Accepted for presentation at IEEE CON 2026