Related papers: How Consistent Are Humans When Grading Programming…
Effective software testing is critical for producing reliable and secure software, yet many computer science students struggle to master the foundational concepts required to construct comprehensive test suites. While automated feedback…
In order to help undergraduate students towards successfully completing their degrees, developing tools that can assist students during the course selection process is a significant task in the education domain. The optimal set of courses…
Many software metrics are designed to measure aspects that are believed to be related to software quality. Static software metrics, e.g., size, complexity and coupling are used in defect prediction research as well as software quality…
Machine-learned systems are in widespread use for making decisions about humans, and it is important that they are fair, i.e., not biased against individuals based on sensitive attributes. We present a general framework of runtime…
Educating students about academic integrity expectations has been suggested as one of the ways to reduce malpractice in take-home programming assignments. We test this hypothesis using data collected from an artificial intelligence course…
Reading code is an essential activity in software maintenance and evolution. Several studies with human subjects have investigated how different factors, such as the employed programming constructs and naming conventions, can impact code…
When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing…
To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer…
Explication and reflection on expert vs. novice considerations within the problem-solving process characterize a cognitive apprenticeship approach for the development of expert-like problem solving practices. In the context of grading, a…
Peer assessment is a popular technique for a more fine-grained evaluation of individual students in group projects. Its effect on the evaluation is well studied. However, its effects on the learning abilities of students are often…
The paper surveys more than forty characterizations of scoring methods for preference aggregation and contains one new result. A general scoring operator is {\it self-consistent} if alternative $i$ is assigned a greater score than $j$…
Developing state-of-the-art approaches for specific tasks is a major driving force in our research community. Depending on the prestige of the task, publishing it can come along with a lot of visibility. The question arises how reliable are…
MOOCs have the potential to revolutionize higher education with their wide outreach and accessibility, but they require instructors to come up with scalable alternates to traditional student evaluation. Peer grading -- having students…
Code review is a critical software engineering practice where developers review code changes before integration to ensure code quality, detect defects, and improve maintainability. In recent years, AI agents that can understand code…
Multiple supervised learning scenarios are composed by a sequence of classification tasks. For instance, multi-task learning and continual learning aim to learn a sequence of tasks that is either fixed or grows over time. Existing…
We investigate whether contemporary multimodal LLMs can assist with grading open-ended calculus at scale without eroding validity. In a large first-year exam, students' handwritten work was graded by GPT-5 against the same rubric used by…
Evaluating student responses, from long essays to short factual answers, is a key challenge in educational NLP. Automated Essay Scoring (AES) focuses on holistic writing qualities such as coherence and argumentation, while Automatic Short…
Despite the growing promise of large language models (LLMs) in automated essay scoring (AES), empirical findings regarding their reliability compared to human raters remain mixed. Following the PRISMA 2020 guidelines, we synthesized 65…
Reflective writing is part of many higher education courses across the globe. It is often considered a challenging task for students as it requires self-regulated learning skills to appropriately plan, timely engage and deeply reflect on…
Although writing code seems trivial at times, problems arise when humans misinterpret what the code actually does. One of the potential causes are "atoms of confusion", the smallest possible patterns of misinterpretable source code.…