English

Why Build an Assistant in Minecraft?

Artificial Intelligence 2019-07-29 v2 Computation and Language
Authors: Arthur Szlam , Jonathan Gray , Kavya Srinet , Yacine Jernite , Armand Joulin , Gabriel Synnaeve , Douwe Kiela , Haonan Yu , Zhuoyuan Chen , Siddharth Goyal , Demi Guo , Danielle Rothermel , C. Lawrence Zitnick , Jason Weston
View on arXiv ↗ PDF ↗

Abstract

In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural language understanding and learning from dialogue.

Cite

@article{arxiv.1907.09273,
  title  = {Why Build an Assistant in Minecraft?},
  author = {Arthur Szlam and Jonathan Gray and Kavya Srinet and Yacine Jernite and Armand Joulin and Gabriel Synnaeve and Douwe Kiela and Haonan Yu and Zhuoyuan Chen and Siddharth Goyal and Demi Guo and Danielle Rothermel and C. Lawrence Zitnick and Jason Weston},
  journal= {arXiv preprint arXiv:1907.09273},
  year   = {2019}
}

Related papers

View all related →
Artificial Intelligence · Computer Science
CraftAssist: A Framework for Dialogue-enabled Interactive Agents
Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu +5
2019-07-22
Human-Computer Interaction · Computer Science
Talking-to-Build: How LLM-Assisted Interface Shapes Player Performance and Experience in Minecraft
Xin Sun, Lei Wang, Yue Li, Jie Li +4
2025-07-31
Computation and Language · Computer Science
Improving Agent Interactions in Virtual Environments with Language Models
Jack Zhang
2024-02-09
Computation and Language · Computer Science
CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant
Yacine Jernite, Kavya Srinet, Jonathan Gray, Arthur Szlam
2019-05-07
Artificial Intelligence · Computer Science
MineStudio: A Streamlined Package for Minecraft AI Agent Development
Shaofei Cai, Zhancun Mu, Kaichen He, Bowei Zhang +3
2025-05-30
Computers and Society · Computer Science
Minecraft: An Engaging Platform to Learn Programming
Worasait Suwannik
2022-08-23
Computation and Language · Computer Science
Large Language Models as Minecraft Agents
Chris Madge, Massimo Poesio
2024-02-14
Computation and Language · Computer Science
Learning to Execute Actions or Ask Clarification Questions
Zhengxiang Shi, Yue Feng, Aldo Lipani
2022-04-22
Multimedia · Computer Science
Deriving Quests from Open World Mechanics
Ryan Alexander, Chris Martens
2017-05-02
Computation and Language · Computer Science
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
Chalamalasetti Kranti, Sherzod Hakimov, David Schlangen
2024-06-26
Computation and Language · Computer Science
A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
Chris Madge, Massimo Poesio
2024-07-18
Artificial Intelligence · Computer Science
Meta-Diversity Search in Complex Systems, A Recipe for Artificial Open-Endedness ?
Mayalen Etcheverry, Bert Wang-Chak Chan, Clément Moulin-Frier, Pierre-Yves Oudeyer
2023-12-04
Machine Learning · Computer Science
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar +6
2022-11-23
Computation and Language · Computer Science
BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
Prashant Jayannavar, Liliang Ren, Marisa Hudspeth, Risham Sidhu +5
2025-09-25
Computation and Language · Computer Science
Generating Instructions at Different Levels of Abstraction
Arne Köhn, Julia Wichlacz, Álvaro Torralba, Daniel Höller +2
2020-10-09
Artificial Intelligence · Computer Science
WHY: Natural Explanations from a Robot Navigator
Raj Korpan, Susan L. Epstein, Anoop Aroor, Gil Dekel
2017-09-29
Artificial Intelligence · Computer Science
EvoCraft: A New Challenge for Open-Endedness
Djordje Grbic, Rasmus Berg Palm, Elias Najarro, Claire Glanois +1
2020-12-10
Human-Computer Interaction · Computer Science
Human-guided Collaborative Problem Solving: A Natural Language based Framework
Harsha Kokel, Mayukh Das, Rakibul Islam, Julia Bonn +9
2022-07-21
Artificial Intelligence · Computer Science
The AI Settlement Generation Challenge in Minecraft: First Year Report
Christoph Salge, Michael Cerny Green, Rodrigo Canaan, Filip Skwarski +5
2021-03-30
Human-Computer Interaction · Computer Science
Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review
Edward Gu, Ho Chit Siu, Melanie Platt, Isabelle Hurley +2
2025-10-01
Artificial Intelligence · Computer Science
Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent
Karolis Jucys, George Adamopoulos, Mehrab Hamidi, Stephanie Milani +6
2024-07-18
Computation and Language · Computer Science
Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft
Sudha Rao, Weijia Xu, Michael Xu, Jorge Leandro +4
2024-07-08
Human-Computer Interaction · Computer Science
Natural Language Interaction to Facilitate Mental Models of Remote Robots
Francisco J. Chiyah Garcia, José Lopes, Helen Hastie
2020-03-13
Programming Languages · Computer Science
Natural Language Specifications in Proof Assistants
Colin S. Gordon, Sergey Matskevich
2022-05-17
Machine Learning · Computer Science
LLaMA Rider: Spurring Large Language Models to Explore the Open World
Yicheng Feng, Yuxuan Wang, Jiazheng Liu, Sipeng Zheng +1
2023-10-16
R2 v1 2026-06-23T10:27:03.438Z