ChipNeMo: Domain-Adapted LLMs for Chip Design
Abstract
ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.
Cite
@article{arxiv.2311.00176,
title = {ChipNeMo: Domain-Adapted LLMs for Chip Design},
author = {Mingjie Liu and Teodor-Dumitru Ene and Robert Kirby and Chris Cheng and Nathaniel Pinckney and Rongjian Liang and Jonah Alben and Himyanshu Anand and Sanmitra Banerjee and Ismet Bayraktaroglu and Bonita Bhaskaran and Bryan Catanzaro and Arjun Chaudhuri and Sharon Clay and Bill Dally and Laura Dang and Parikshit Deshpande and Siddhanth Dhodhi and Sameer Halepete and Eric Hill and Jiashang Hu and Sumit Jain and Ankit Jindal and Brucek Khailany and George Kokai and Kishor Kunal and Xiaowei Li and Charley Lind and Hao Liu and Stuart Oberman and Sujeet Omar and Ghasem Pasandi and Sreedhar Pratty and Jonathan Raiman and Ambar Sarkar and Zhengjiang Shao and Hanfei Sun and Pratik P Suthar and Varun Tej and Walker Turner and Kaizhe Xu and Haoxing Ren},
journal= {arXiv preprint arXiv:2311.00176},
year = {2024}
}
Comments
Updated results for ChipNeMo-70B model