Related papers: SIFBench: An Extensive Benchmark for Fatigue Analy…

Predicting protein complex structures is essential for protein function analysis, protein design, and drug discovery. While AI methods like AlphaFold can predict accurate structural models for many protein complexes, reliably estimating the…

Safety Instrumented Systems (SIS) protect major hazard facilities, e.g. power plants, against catastrophic accidents. An SIS consists of hardware components and a controller software -- the ``program''. Current safety analyses of SIS'…

Effective decision-making often relies on identifying what makes each candidate distinctive. While existing benchmarks for LLMs emphasize retrieving or summarizing information relevant to a given query, they do not evaluate a model's…

Automated analysis for engineering structures offers considerable potential for boosting efficiency by minimizing repetitive tasks. Although AI-driven methods are increasingly common, no systematic framework yet leverages Large Language…

Due to the varying intensity of pavement cracks, the complexity of topological structure, and the noise of texture background, image classification for asphalt pavement cracking has proven to be a challenging problem. Fatigue cracking, also…

LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime…

The broad application of polymer composites in engineering demands the deep understanding of the main damage mechanisms under realistic loading conditions and the development of proper physics-based models. Towards this goal, this study…

Software vulnerabilities can cause numerous problems, including crashes, data loss, and security breaches. These issues greatly compromise quality and can negatively impact the market adoption of software applications and systems.…

As AI-driven document understanding and processing tools become increasingly prevalent in real-world applications, the need for rigorous evaluation standards has grown increasingly urgent. Existing benchmarks and evaluations often focus on…

As models become increasingly sophisticated, conventional algorithm benchmarks are increasingly saturated, underscoring the need for more challenging benchmarks to guide future improvements in algorithmic reasoning. This paper introduces…

Fault diagnosis of mechanical equipment involves data collection, feature extraction, and pattern recognition but is often hindered by the imbalanced nature of industrial data, introducing significant uncertainty and reducing diagnostic…

We present MaterialFigBench, a benchmark dataset designed to evaluate the ability of multimodal large language models (LLMs) to solve university-level materials science problems that require accurate interpretation of figures. Unlike…

Crack initiation governs high cycle fatigue life and is susceptible to microstructural details. While corresponding microstructure-sensitive models are available, their validation is difficult. We propose a validation framework where a…

The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following. Yet, their effectiveness often diminishes in…

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that…

The rapid adoption of machine learning (ML) in domain sciences necessitates best practices and standardized benchmarking for performance evaluation. We present Matbench Discovery, an evaluation framework for ML energy models, applied as…

Context: Recent software engineering (SE) research has highlighted the need for sociotechnical research, implying a demand for customized psychometric scales. Objective: We define the concepts of technical and sociotechnical infrastructure…

We introduce, a large-scale synthetic benchmark of 15,045 university-level physics problems (90/10% train/test split). Each problem is fully parameterized, supporting an effectively infinite range of input configurations, and is accompanied…

The plastic events occurring during the process of intergranular fracture in metals is still not well understood due to the complexity of grain boundary (GB) structures and their interactions with crack-tip dislocation plasticity. By…

Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation…