Global empirical risk minimizers with "shape constraints" are rate optimal in general dimensions
share
Entropy integrals are widely used as a powerful tool to obtain upper bounds for the rates of convergence of global empirical risk minimizers (ERMs), in standard settings such as density estimation and regression. The upper bound for the convergence rates thus obtained typically matches the minimax lower bound when the entropy integral converges, but admits a strict gap compared with the lower bound when it diverges. [BM93] provided a striking example showing that such a gap is real with the entropy structure alone: for a variant of the natural Hölder class with low regularity, the global ERM actually converges at the rate predicted by the entropy integral that substantially deviates from the lower bound. The counter-example has spawned a long-standing negative position on the use of global ERMs in the regime where the entropy integral diverges, as they are heuristically believed to converge at a sub-optimal rate in a variety of models. The present paper demonstrates that this gap can be closed if the models admit certain degree of `shape constraints' in addition to the entropy structure. In other words, the global ERMs in such `shape-constrained' models will indeed be rate-optimal, matching the lower bound even when the entropy integral diverges. The models with `shape constraints' we investigate include (i) edge estimation with additive and multiplicative errors, (ii) binary classification, (iii) multiple isotonic regression, (iv) s-concave density estimation, all in general dimensions when the entropy integral diverges. Here `shape constraints' are interpreted broadly in the sense that the complexity of the underlying models can be essentially captured by the size of the empirical process over certain class of measurable sets, for which matching upper and lower bounds are obtained to facilitate the derivation of sharp convergence rates for the associated global ERMs.
READ FULL TEXT