Three Tales of Mathematics of Machine Learning

Abstract: I will tell three mathematical tales of machine learning related to my most recent work: 1. identification of deep neural
networks, 2. global optimization over manifolds, 3. Mean-field optimal control of NeurODE. Tale 1. is about the proof that, despite the
NP-hardness of the problem, generic neural networks can be identified up to natural symmetries by a finite number of input-output samples scaling
with the complexity of the network. Numerical validation of the result is presented. A crucial subproblem of the identification pipeline is the
solution of a nonconvex optimization over the sphere. Tale 2. is in fact about solving global optimizations over spheres by means of a
multi-agent dynamics, which combines a consensus mechanism and random exploration. The proof of global solution is based on showing that the
large particle limit of the SDE system is distributed as the solution of the deterministic PDE, whose large time asymptotics converges to a
global minimizer. I present numerical results in robust linear regression for computing eigenfaces. In the Tale 3. I introduce NeurODE,
which are neural networks approximable by ODE. I show that their training can be formulated as a mean-field optimal control and I present
the derivation of a mean-field Pontryagin maximum principle characterizing optimal parameters/controls and its well-posedness. Again
a numerical experiment of a simple 2D classification problem validates the theoretical results.