Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation, plus a standardized JSON schema and tooling for VLM and human-based judging. Across the evaluated models, per-attempt pass rates span 34-83 percent and effective cost per success spans USD 0.66-1.42. Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews.
Cite
@article{arxiv.2602.00105,
title = {HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models},
author = {Wing Chan and Richard Allen},
journal= {arXiv preprint arXiv:2602.00105},
year = {2026}
}
Comments
14 pages, 5 figures, for code and data, see https://github.com/sourceful-official/hype-edit-1-benchmark