New Step-Size Criterion for the Steepest Descent based on Geometric Numerical Integration
share
This paper deals with unconstrained optimization problems based on numerical analysis of ordinary differential equations (ODEs). Although it has been known for a long time that there is a relation between optimization methods and discretization of ODEs, research in this direction has recently been gaining attention. In recent studies, the dissipation laws of ODEs have often played an important role. By contrast, in the context of numerical analysis, a technique called geometric numerical integration, which explores discretization to maintain geometrical properties such as the dissipation law, is actively studied. However, in research investigating the relationship between optimization and ODEs, techniques of geometric numerical integration have not been sufficiently investigated. In this paper, we show that a recent geometric numerical integration technique for gradient flow reads a new step-size criterion for the steepest descent method. Consequently, owing to the discrete dissipation law, convergence rates can be proved in a form similar to the discussion in ODEs. Although the proposed method is a variant of the existing steepest descent method, it is suggested that various analyses of the optimization methods via ODEs can be performed in the same way after discretization using geometric numerical integration.
READ FULL TEXT