English

Solver-Hard Is Not Model-Hard: A Hardness-Controlled Diagnostic for LLM Constraint Reasoning

Machine Learning 2026-07-19 v1 Artificial Intelligence Logic in Computer Science

Abstract

LLM constraint reasoners are often evaluated near the random-SAT phase transition, confounding density and solver hardness. We test instance-level transfer while near-matching clause density. At aligned size bins, with near-matched density and matched maximum clause width, we compare proof-hard expander-Tseitin and proof-easy ladder-Tseitin formulas, pigeonhole anchors, and density-mismatched controls. Theory separates their resolution hardness; a solver-specific Glucose mean-conflict proxy differs by up to 51×51\times, and five other solvers preserve the direction. Across three included models (243 instances each; a fourth is excluded for abstention), the near-matched-density accuracy gaps range from 32-32 to +20+20 points, with a pooled gap of +1.7+1.7 points (p=0.74p=0.74) and a wrong-signed correctness-versus-conflict association (r=+0.15r=+0.15). A proof-preserving relabeling lowers accuracy in all five clusters for one model (mean 93-93 points) but not another, exposing model-surface sensitivity. In a preregistered extension, provider-reported completion-token spend does not consistently increase with the proxy after accounting for formula length and censoring. At 16k, the reasoning model spends more on proof-easy matched formulas and exhausts its budget on the solver-easiest UNSAT family; the 32k C1 gap is absent. These scoped dissociations concern verdict accuracy and observed token spend, not certificate solving, exact proof length, or allocation efficiency.

Keywords

Cite

@article{arxiv.2607.17047,
  title  = {Solver-Hard Is Not Model-Hard: A Hardness-Controlled Diagnostic for LLM Constraint Reasoning},
  author = {Lucky Verma},
  journal= {arXiv preprint arXiv:2607.17047},
  year   = {2026}
}

Comments

13 pages, 2 figures, 5 tables. Code and aggregate reproduction data: https://github.com/lucky-verma/solver-hard-is-not-model-hard