Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search
Abstract
In this work we consider Code World Models, world models generated by a Large Language Model (LLM) in the form of Python code for model-based Reinforcement Learning (RL). Calling code instead of LLMs for planning has potential to be more precise, reliable, interpretable, and extremely efficient. However, writing appropriate Code World Models requires the ability to understand complex instructions, to generate exact code with non-trivial logic and to self-debug a long program with feedback from unit tests and environment trajectories. To address these challenges, we propose Generate, Improve and Fix with Monte Carlo Tree Search (GIF-MCTS), a new code generation strategy for LLMs. To test our approach in an offline RL setting, we introduce the Code World Models Benchmark (CWMB), a suite of program synthesis and planning tasks comprised of 18 diverse RL environments paired with corresponding textual descriptions and curated trajectories. GIF-MCTS surpasses all baselines on the CWMB and two other benchmarks, and we show that the Code World Models synthesized with it can be successfully used for planning, resulting in model-based RL agents with greatly improved sample efficiency and inference speed.
Cite
@article{arxiv.2405.15383,
title = {Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search},
author = {Nicola Dainese and Matteo Merler and Minttu Alakuijala and Pekka Marttinen},
journal= {arXiv preprint arXiv:2405.15383},
year = {2024}
}
Comments
Accepted at NeurIPS 2024, Main Track. 11 pages in main text, 40 pages including references and supplementary materials. 2 figures and 3 tables in the main text, 9 figures and 12 tables when including the supplementary materials. Website at https://sites.google.com/view/code-world-models/home