Review of attention mechanisms in deep neural networks

12 marzo 2019, 14:00
Versione stampabile

Date & Time: March 12, h.14:00 
Location: Room Garda, Polo scientifico e tecnologico "Fabio Ferrari", Building Povo 1, via Sommarive 5, Povo (Trento)


  • Prof. Renato De Mori, McGIll University 


Recent encoder/decoder neural architectures with recurrent neural networks (RNN) including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) elements have been used with impressive performance. Nevertheless, the inherently sequential nature of RNNs precludes parallelization within training examples. This makes it difficult to identify useful context features in long sequences of data, as memory constraints limit batching across examples, potentially ignoring the structured information typical of natural language sequences. To alleviate these problems, attention mechanisms have been introduced for modeling dependencies without regard to distance in the input or output related data. These mechanisms are based on the notion of attention as a non-uniform spatial distribution of relevant features, and the explicit scalar representation of their relative relevance, Mechanisms of attention have become almost a de facto standard in in tasks such as multi language translation, image and video captioning, question answering, language modeling, machine comprehension, relation extraction, and others. 

Motivations and essential components of a selection of mechanisms proposed between 2014 and 2018 will be reviewed. It includes self-attention, transformer, structured attention, multi-scale attention, semantic attention, compositional attention, graph attention, and learn to pay attention

About the Speaker

Renato De Mori is emeritus professor at Mc Gill University (Canada) and at the University of Avignon (France). He has been full professor in Italy (University of Turin), Canada and France. He is a Life Fellow of the Computer Society and has been distinguished lecturer of the Signal Processing Society of the Institute of Electrical and Electronic Engineers (IEEE). 

More details: Renato De Mori

Contact: giuseppe.riccardi [at] (Giuseppe Riccardi)