Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models

Highlights：

• How to build a cooperative mechanism in emotion recognition to integrate the advantages of the two models has been ignored for a long time. By taking advantage of both discrete and dimensional perspectives on emotion, we introduce an exploration and exploitation mechanism, which means the proposed emotion recognition model not only can accurately locate a simple and discrete emotional anchor in the whole continuous emotion space (exploration) but can also effectively search for complex and subtle emotional states near the emotional anchor (exploitation). Specifically, we propose a multitask graph-neural network model to incorporate this mechanism. To our knowledge, our method is to tackle the synergy problem of emotion analysis models based on graph neural networks firstly from the perspective of discrete and dimensional models.

• In addition, we conduct experiments on AVEC, MELD, and IEMOCAP datasets respectively. The experimental results show that achieves better results than a number of state-of-the-art methods in the application of emotion classification.

摘要

•Both discrete and dimensional models can be applied to emotion recognition in psychology or computer science. In view of their different definitions, they have certain differences in emotion recognition tasks. Based on dimensional models of emotion, a certain emotion can estimate the differences and similarities with other emotions easily thanks to the continuity of the numerical vector. On the other hand, the core emotions based on discrete models have better understandability, which is biologically determined emotional responses and their expression and recognition are basically the same for all individuals regardless of cultural or ethnic differences.•How to build a cooperative mechanism in emotion recognition to integrate the advantages of the two models has been ignored for a long time. By taking advantage of both discrete and dimensional perspectives on emotion, we introduce an exploration and exploitation mechanism, which means the proposed emotion recognition model not only can accurately locate a simple and discrete emotional anchor in the whole continuous emotion space (exploration) but can also effectively search for complex and subtle emotional states near the emotional anchor (exploitation). Specifically, we propose a multitask graph-neural network model to incorporate this mechanism. To our knowledge, our method is to tackle the synergy problem of emotion analysis models based on graph neural networks firstly from the perspective of discrete and dimensional models.•In addition, we conduct experiments on AVEC, MELD, and IEMOCAP datasets respectively. The experimental results show that achieves better results than a number of state-of-the-art methods in the application of emotion classification.