Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG.
@article{arxiv.2409.01344,
title = {Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A},
author = {K Roth and Rushil Gupta and Simon Halle and Bang Liu},
journal= {arXiv preprint arXiv:2409.01344},
year = {2025}
}