Related papers: Logical coherence in Bayesian simultaneous three-w…
In this paper, we consider Bayesian hypothesis testing for the balanced one-way random effects model. A special choice of the prior formulation for the ratio of variance components is shown to yield an explicit closed-form Bayes factor…
Although logical consistency is desirable in scientific research, standard statistical hypothesis tests are typically logically inconsistent. In order to address this issue, previous work introduced agnostic hypothesis tests and proved that…
Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow…
This paper explores conditions of existence of different types of consistent tests. New links of these types of consistency are also established. The existence of discernible (strong consistent) tests follows from the existence of pointwise…
In Bayesian statistics the precise point-null hypothesis $\theta=\theta_0$ can be tested by checking whether $\theta_0$ is contained in a credible set. This permits testing of $\theta=\theta_0$ without having to put prior probabilities on…
In this paper, we propose a general method for testing composite hypotheses. Our idea is to use confidence limits to define stopping and decision rules. The requirements of operating characteristic function can be satisfied by adjusting the…
Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency. We argue that self-consistency is an important…
Scientific claims gain credibility by replicability, especially if replication under different circumstances and varying designs yields equivalent results. Aggregating results over multiple studies is, however, not straightforward, and when…
Do larger and more capable language models learn to update their "beliefs" about propositions more consistently with Bayes' theorem when presented with evidence in-context? To test this, we formulate a Bayesian Coherence Coefficient (BCC)…
This paper proposes CES, a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness of variable predictions during execution simulation, CES…
Autoregressive Large Language Models (LLMs) trained for next-word prediction have demonstrated remarkable proficiency at producing coherent text. But are they equally adept at forming coherent probability judgments? We use probabilistic…
In the modern Bayesian view classical probability theory is simply an extension of conventional logic, i.e., a quantitative tool that allows for consistent reasoning in the presence of uncertainty. Classical theory presupposes, however,…
It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings…
Given the well-known and fundamental problems with hypothesis testing via classical (point-form) significance tests, there has been a general move to alternative approaches, often focused on the Bayesian t-test. We show that the Bayesian…
It is well known that a Bayesian probability forecast for all future observations should be a probability measure in order to satisfy a natural condition of coherence. The main topics of this paper are the evolution of the Bayesian…
Test log-likelihood is commonly used to compare different models of the same data or different approximate inference algorithms for fitting the same probabilistic model. We present simple examples demonstrating how comparisons based on test…
We review and test twelve different approaches to the detection of finite-time coherent material structures in two-dimensional, temporally aperiodic flows. We consider both mathematical methods and diagnostic scalar fields, comparing their…
Large language models (LLMs) have achieved remarkable successes on various tasks. However, recent studies have found that there are still significant challenges to the logical reasoning abilities of LLMs, which can be categorized into the…
In this paper, we prove that separable negative log-likelihood losses for structured prediction are not necessarily Bayes consistent, or, in other words, minimizing these losses may not result in a model that predicts the most probable…
Analogues of the frequentist chi-square and F tests are proposed for testing goodness-of-fit and consistency for Bayesian models. Simple examples exhibit these tests' detection of inconsistency between consecutive experiments with identical…