Additive noise annealing and approximation properties of quantized neural networks

Luogo: Povo Zero, via Sommarive, 14 (Povo) – Sala Seminari "-1"
Ore: 15:00  

Relatore:

• Matteo Spallanzani (Università di Modena e Reggio Emilia)    

Abstract:

Much of the data we collect and analyse is highly complex, and no closed-form formulas are available that can be translated into algorithms to process it. In such scenarios, machine learning (ML) models come to help. Deep neural networks (DNNs) are brain-inspired computational models which have proven to be extremely useful and flexible. DNNs typically use millions or billions of parameters to encode complex non-linear functions and perform as many operations to perform inference on a single data point. Therefore, deploying DNNs on resource-constrained computers designed to process data directly at its source (edge computing) is a challenging problem. In this seminar, we will introduce quantized neural networks (QNNs), a sub-class of DNNs which uses discontinuous activation functions and parameters taken from finite sets. First, we will analyse the approximation capabilities of QNNs. Then, we will show how adding noise to the argument of step functions can regularize them to yield differentiable functions. Finally, we will present additive noise annealing (ANA), a gradient-based algorithm designed to train QNNs inspired by this regularization result.

Referente: Gian Paolo Leonardi