Seminar

A two-scale complexity measure for stochastic neural networks

Seminario periodico del Dipartimento di Matematica
30 November 2023
Start time 
2:30 pm
PovoZero - Via Sommarive 14, Povo (Trento)
Aula Seminari 1– Povo0 e online Zoom (contattare dept.math@unitn.it per le credenziali)
Target audience: 
University community
UniTrento students
Attendance: 
Free
Online
Registration email: 
Contact person: 
Prof. Gian Paolo Leonardi
Contact details: 
Università degli Studi Trento 38123 Povo (TN) - Staff Dipartimento di Matematica
+39 0461/281508-1625-1701-1980-3898
Speaker: 
Massimiliano Datres (Università di Trento)

Abstract

Over-parametrized deep learning models are achieving outstanding performances in solving several complex tasks such as image classification problems, object detection, and natural language processing. Despite the risk of overfitting, these parametric models show impressive generalization after training. Hence, defining appropriate complexity measures becomes crucial for understanding and quantifying the generalization capabilities of deep learning models. In this talk, I will introduce a new notion of complexity measure, called two-scale effective dimension (2sED), which is a box-covering dimension related to a metric induced by the Fisher information matrix of the parametric model. I will then show how the 2sED can be used to derive a generalization bound. Furthermore, I present an approximation of the 2sED for Markovian models, called lower 2sED, that can be computed sequentially layer-by-layer with less computational demands. Finally, I present experimental evidence that the post-training performance of given parametric models is related both with 2sED and the lower 2sED.