Related papers: General Mathematical Proof of Occam's Razor; Upgra…
Mathematical models are an essential component of quantitative science. They generate predictions about the future, based on information available in the present. In the spirit of Occam's razor, simpler is better; should two models make…
Statistical learning theory is often associated with the principle of Occam's razor, which recommends a simplicity preference in inductive inference. This paper distills the core argument for simplicity obtainable from statistical learning…
We provide a new representation-independent formulation of Occam's razor theorem, based on Kolmogorov complexity. This new formulation allows us to: (i) Obtain better sample complexity than both length-based and VC-based versions of Occam's…
This paper presents new experimental evidence against the utility of Occam's razor. A~systematic procedure is presented for post-processing decision trees produced by C4.5. This procedure was derived by rejecting Occam's razor and instead…
Ensemble theories have received a lot of interest recently as a means of explaining a lot of the detailed complexity observed in reality by a vastly simpler description ``every possibility exists'' and a selection principle ({\em Anthropic…
The task of parametric model selection is cast in terms of a statistical mechanics on the space of probability distributions. Using the techniques of low-temperature expansions, we arrive at a systematic series for the Bayesian posterior…
A formal theory of simplicity is introduced, in the context of a "combinational" computation model that views computation as comprising the iterated transformational and compositional activity of a population of agents upon each other.…
Occam's Razor tells us to pick the simplest model that fits our observations. In order to make sense of his process mathematically, we interpret it in the context of posets of functions. Our approach leads to some unusual new combinatorial…
Prompted by misconceptions in the recent literature, we review the justifications for naturalness arguments and Occam's razor found in Bayesian statistics. We discuss the automatic Occam's razor that emerges in Bayesian formalism, bringing…
The algorithms available for retail forecasting have increased in complexity. Newer methods, such as machine learning, are inherently complex. The more traditional families of forecasting models, such as exponential smoothing and…
We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's…
Non-deductive reasoning, encompassing inductive and abductive reasoning, is essential in addressing complex real-world questions. One key feature of inductive and abductive reasoning is that there are many valid hypotheses; the simplest…
Ockham's razor is a heuristic concept applied in philosophy of science to decide between two or more feasible physical theories. Ockham's razor operates by deciding in favour of the theory with least assumptions and concepts; roughly…
Lately there has been a lot of discussion about why deep learning algorithms perform better than we would theoretically suspect. To get insight into this question, it helps to improve our understanding of how learning works. We explore the…
This article applies the principle of Occam's Razor to non-parametric model building of statistical data, by finding a model with the minimal number of bits, leading to an exceptionally effective regularization method for probability…
Scientists have long preferred the simplest possible explanation of their data. More re-cently, a worrying trend to favor complex interpretations has taken hold because they are perceived as more impactful.
Most traditional artificial intelligence (AI) systems of the past 50 years are either very limited, or based on heuristics, or both. The new millennium, however, has brought substantial progress in the field of theoretically optimal and…
Bayesian inference provides a uniquely rigorous approach to obtain principled justification for uncertainty in predictions, yet it is difficult to articulate suitably general prior belief in the machine learning context, where computational…
Here we argue that the notion of falsifiability, a key concept in defining a valid scientific theory, can be quantified using Bayesian Model Selection, which is a standard tool in modern statistics. This relates falsifiability to the…
In real-life statistical data, it seems that conditional probabilities for the effect given their causes tend to be less complex and smoother than conditionals for causes, given their effects. We have recently proposed and tested methods…