Related papers: An Elementary Proof of the Near Optimality of LogS…
Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to…
The LogSumExp function, dual to the Kullback-Leibler (KL) divergence, plays a central role in many important optimization problems, including entropy-regularized optimal transport (OT) and distributionally robust optimization (DRO). In…
This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex. We show that no algorithm can reach an error $\epsilon$ in minimizing all functions from…
We consider the optimal control of a differential equation that involves the suprema of the state over some part of the history. In many applications, this non-smooth functional dependence is crucial for the successful modeling of…
A soft-max function has two main efficiency measures: (1) approximation - which corresponds to how well it approximates the maximum function, (2) smoothness - which shows how sensitive it is to changes of its input. Our goal is to identify…
We propose a general method for optimization with semi-infinite constraints that involve a linear combination of functions, focusing on the case of the exponential function. Each function is lower and upper bounded on sub-intervals by…
We study the elliptic maximal functions defined by averages over ellipses and rotated ellipses which are multi-parametric variants of the circular maximal function. We prove that those maximal functions are bounded on $L^p$ for some $p\neq…
An adaptive regularization algorithm using inexact function and derivatives evaluations is proposed for the solution of composite nonsmooth nonconvex optimization. It is shown that this algorithm needs at most…
Let us assume that $f$ is a continuous function defined on the unit ball of $\mathbb R^d$, of the form $f(x) = g (A x)$, where $A$ is a $k \times d$ matrix and $g$ is a function of $k$ variables for $k \ll d$. We are given a budget $m \in…
Several new inequalities for moduli of smoothness and errors of the best approximation of a function and its derivatives in the spaces $L_p$, $0<p<1$, are obtained. For example, it is shown that for any $0<p<1$ and $k,\,r\in \mathbb{N}$ one…
Smooth finite-sum optimization has been widely studied in both convex and nonconvex settings. However, existing lower bounds for finite-sum optimization are mostly limited to the setting where each component function is (strongly) convex,…
We study the classical problem of approximating a non-decreasing function $f: \mathcal{X} \to \mathcal{Y}$ in $L^p(\mu)$ norm by sequentially querying its values, for known compact real intervals $\mathcal{X}$, $\mathcal{Y}$ and a known…
We derive optimal rates of convergence in the supremum norm for estimating the H\"older-smooth mean function of a stochastic process which is repeatedly and discretely observed with additional errors at fixed, multivariate, synchronous…
This paper considers the problem of smoothing convex functions and sets, seeking the nearest smooth convex function or set to a given one. For convex cones and sublinear functions, a full characterization of the set of all optimal…
We propose novel smooth approximations to the classical rounding function, suitable for differentiable optimization and machine learning applications. Our constructions are based on two approaches: (1) localized sigmoid window functions…
Mathematical morphology is a part of image processing that has proven to be fruitful for numerous applications. Two main operations in mathematical morphology are dilation and erosion. These are based on the construction of a supremum or…
We seek to improve the pooling operation in neural networks, by applying a more theoretically justified operator. We demonstrate that LogSumExp provides a natural OR operator for logits. When one corrects for the number of elements inside…
The optimal function $f$ satisfying $$ \mathbb{E} |\sum_{1}^n X_i | \ge f(\mathrbb{E}|X_1|,...,\mathbb{E}|X_n|) $$ for every martingale $(X_1,X_1+X_2, ...,\sum_{i=1}^n X_i)$ is shown to be given by $$ f(a) = \max \Big\{a_k-\sum_{i=1}^{k-1}…
The purpose of the paper is to provide a characterization of the error of the best polynomial approximation of composite functions in weighted spaces. Such a characterization is essential for the convergence analysis of numerical methods…
Several results on constrained spline smoothing are obtained. In particular, we establish a general result, showing how one can constructively smooth any monotone or convex piecewise polynomial function (ppf) (or any $q$-monotone ppf,…