AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations

Jan Hartman; Rishabh Mehrotra; Hitesh Sagtani; Dominic Cooney; Rafal Gajdulewicz; Beyang Liu; Julie Tibshirani; Quinn Slack

doi:10.1145/3640457.3688060

AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations

Information Retrieval 2024-08-13 v1 Machine Learning Software Engineering

Authors: Jan Hartman , Rishabh Mehrotra , Hitesh Sagtani , Dominic Cooney , Rafal Gajdulewicz , Beyang Liu , Julie Tibshirani , Quinn Slack

View on arXiv ↗ PDF ↗ DOI ↗

Abstract

In this work, we discuss a recently popular type of recommender system: an LLM-based coding assistant. Connecting the task of providing code recommendations in multiple formats to traditional RecSys challenges, we outline several similarities and differences due to domain specifics. We emphasize the importance of providing relevant context to an LLM for this use case and discuss lessons learned from context enhancements & offline and online evaluation of such AI-assisted coding systems.

Keywords

code generation recommender system in-context learning

Cite

@article{arxiv.2408.05344,
  title  = {AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations},
  author = {Jan Hartman and Rishabh Mehrotra and Hitesh Sagtani and Dominic Cooney and Rafal Gajdulewicz and Beyang Liu and Julie Tibshirani and Quinn Slack},
  journal= {arXiv preprint arXiv:2408.05344},
  year   = {2024}
}

AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations

Abstract

Keywords

Cite

Related papers