Related papers: Direct Multi-Step Forecast based Comparison of Nes…

Comparing competing mathematical models of complex natural processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for…

Nested nonparametric processes are vectors of random probability measures widely used in the Bayesian literature to model the dependence across distinct, though related, groups of observations. These processes allow a two-level clustering,…

We propose a general method for distributed Bayesian model choice, using the marginal likelihood, where a data set is split in non-overlapping subsets. These subsets are only accessed locally by individual workers and no data is shared…

In this paper, we propose a general method for testing composite hypotheses. Our idea is to use confidence limits to define stopping and decision rules. The requirements of operating characteristic function can be satisfied by adjusting the…

As predictive algorithms grow in popularity, using the same dataset to both train and test a new model has become routine across research, policy, and industry. Sample-splitting attains valid inference on model properties by using separate…

Time series forecasting using historical data has been an interesting and challenging topic, especially when the data is corrupted by missing values. In many industrial problem, it is important to learn the inference function between the…

We investigate model assessment and selection in a changing environment, by synthesizing datasets from both the current time period and historical epochs. To tackle unknown and potentially arbitrary temporal distribution shift, we develop…

This paper provides a review of model selection and model averaging methods for multinomial probit models estimated using the MACML approach. The proposed approaches are partitioned into test based methods (mostly derived from the…

Assessing the validity of a real-world system with respect to given quality criteria is a common yet costly task in industrial applications due to the vast number of required real-world tests. Validating such systems by means of simulation…

Cross-validation is a widely used technique for evaluating the performance of prediction models, ranging from simple binary classification to complex precision medicine strategies. It helps correct for optimism bias in error estimates,…

A rich set of frequentist model averaging methods has been developed, but their applications have largely been limited to point prediction, as measuring prediction uncertainty in general settings remains an open problem. In this paper we…

In model checking for regressions, nonparametric estimation-based tests usually have tractable limiting null distributions and are sensitive to oscillating alternative models, but suffer from the curse of dimensionality. In contrast,…

The problem of multi-hypothesis testing with controlled sensing of observations is considered. The distribution of observations collected under each control is assumed to follow a single-parameter exponential family distribution. The goal…

We describe a new method for evaluating Bayes factors. The key idea is to introduce a hypermodel in which the competing models are components of a mixture distribution. Inference for the mixing probabilities then yields estimates of the…

System modeling is a classical approach to ensure their reliability since it is suitable both for a formal verification and for software testing techniques. In the context of model-based testing an approach combining random testing and…

In this paper we discuss how to evaluate the differences between fitted logistic regression models across sub-populations. Our motivating example is in studying computerized diagnosis for learning disabilities, where sub-populations based…

When machine learning systems meet real world applications, accuracy is only one of several requirements. In this paper, we assay a complementary perspective originating from the increasing availability of pre-trained and regularly…

Mathematical programming formulations of influence diagrams can bridge the gap between representing and solving decision problems. However, they suffer from both modeling and computational limitations. Aiming to address modeling…

We consider probabilistic multinomial probit classification using Gaussian process (GP) priors. The challenges with the multiclass GP classification are the integration over the non-Gaussian posterior distribution, and the increase of the…

In this paper, we introduce a flexible and widely applicable nonparametric entropy-based testing procedure that can be used to assess the validity of simple hypotheses about a specific parametric population distribution. The testing…