Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach
Abstract
A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.
Cite
@article{arxiv.2407.12687,
title = {Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach},
author = {Irina Jurenka and Markus Kunesch and Kevin R. McKee and Daniel Gillick and Shaojian Zhu and Sara Wiltberger and Shubham Milind Phal and Katherine Hermann and Daniel Kasenberg and Avishkar Bhoopchand and Ankit Anand and Miruna Pîslar and Stephanie Chan and Lisa Wang and Jennifer She and Parsa Mahmoudieh and Aliya Rysbek and Wei-Jen Ko and Andrea Huber and Brett Wiltshire and Gal Elidan and Roni Rabin and Jasmin Rubinovitz and Amit Pitaru and Mac McAllister and Julia Wilkowski and David Choi and Roee Engelberg and Lidan Hackmon and Adva Levin and Rachel Griffin and Michael Sears and Filip Bar and Mia Mesar and Mana Jabbour and Arslan Chaudhry and James Cohan and Sridhar Thiagarajan and Nir Levine and Ben Brown and Dilan Gorur and Svetlana Grant and Rachel Hashimshoni and Laura Weidinger and Jieru Hu and Dawn Chen and Kuba Dolecki and Canfer Akbulut and Maxwell Bileschi and Laura Culp and Wen-Xin Dong and Nahema Marchal and Kelsie Van Deman and Hema Bajaj Misra and Michael Duah and Moran Ambar and Avi Caciularu and Sandra Lefdal and Chris Summerfield and James An and Pierre-Alexandre Kamienny and Abhinit Mohdi and Theofilos Strinopoulous and Annie Hale and Wayne Anderson and Luis C. Cobo and Niv Efron and Muktha Ananda and Shakir Mohamed and Maureen Heymans and Zoubin Ghahramani and Yossi Matias and Ben Gomes and Lila Ibrahim},
journal= {arXiv preprint arXiv:2407.12687},
year = {2025}
}