AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
@article{arxiv.2302.07427,
title = {Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming},
author = {Majeed Kazemitabaar and Justin Chow and Carl Ka To Ma and Barbara J. Ericson and David Weintrop and Tovi Grossman},
journal= {arXiv preprint arXiv:2302.07427},
year = {2023}
}
Comments
To be published in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), April 23--28, 2023, Hamburg, Germany 17 pages with 11 Figures, 2 Tables, 6 Page Appendix