Related papers: Rubric-based holistic review represents a change f…

As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may…

A recent paper evaluating a new rubric-based graduate admissions approach using generic methods tentatively suggested that its decisions differed noticeably from the previous approach in an unspecified way. Using prior knowledge that the…

Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of…

Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions…

End-of-course assessments play important roles in the ongoing attempt to improve instruction in physics courses. Comparison of students' performance on assessments before and after instruction gives a measure of student learning. In…

Teaching assistants (TAs) are often responsible for grading in introductory physics courses at large research universities. Their grading practices can shape students' approaches to problem solving and learning. Physics education research…

This work aims to understand how effective the typical admissions criteria used in physics are at identifying students who will complete the PhD. Through a multivariate statistical analysis of a sample that includes roughly one in eight…

Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams, creating substantial variation in format and interpretation. Despite their importance for evaluating students'…

This article addresses the challenge of validating the admission committee's decisions for undergraduate admissions. In recent years, the traditional review process has struggled to handle the overwhelmingly large amount of applicants'…

Rubrics have been extensively utilized for evaluating unverifiable, open-ended tasks, with recent research incorporating them into reward systems for reinforcement learning. However, existing frameworks typically treat rubrics only as…

Conventional reward modeling relies on gradient descent over neural weights, creating opaque, data-hungry "black boxes." We propose a paradigm shift from implicit to explicit reward parameterization, recasting optimization from continuous…

Scalar reward models compress multi-dimensional human preferences into a single opaque score, creating an information bottleneck that often leads to brittleness and reward hacking in open-ended alignment. We argue that robust alignment for…

Statistical decision algorithms are increasingly deployed in domains where ground-truth labels are hard to obtain, such as hiring, university admissions, and content moderation. In these settings, models are typically trained on historical…

While large language models (LLMs) have been used for automated grading, they have not yet achieved the same level of performance as humans, especially when it comes to grading complex questions. Existing research on this topic focuses on a…

The open-ended generation in LLMs usually requires multi-dimensional rubrics to adequately assess quality and guide the improvement of reinforcement learning. However, a critical dilemma inherent in this training paradigm is the imbalanced…

A growing number of college applications has presented an annual challenge for college admissions in the United States. Admission offices have historically relied on standardized test scores to organize large applicant pools into viable…

Algorithmic decision systems are increasingly used in areas such as hiring, school admission, or loan approval. Typically, these systems rely on labeled data for training a classification model. However, in many scenarios, ground-truth…

We use self-reported data from 2011--2014 and determine the academic profile of accepted students at physics graduate programs in the United States. We analyze the accepted students' grade point averages and physics Graduate Record…

Rubrics are being used in a wide variety of disciplines in higher education to evaluate assessments and provide feedback to students. Rubrics are traditionally implemented as paper-based table format to grade assessments and provide…

Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards…