wellton/Multimodal-Emotion-Recognition
1
0
Fork 0
Código Issues Pull Requests Ações Pacotes Projetos Versões Wiki Atividade

Newviz

Ver Código-Fonte
Esse commit está contido em:
Maël
2019-06-25 12:18:17 +02:00
pai f27fbe98e8 f5d77ca6bf
commit 149bbf559b
21 arquivos alterados com 132 adições e 17 exclusões
Baixar Arquivo de Patch Baixar Arquivo de Diff Expandir todos os arquivos Colapsar todos os arquivos
00-Presentation/Reports/PoleEmploi_Short_with_WebPage.pdf
BIN
Ver Arquivo
Arquivo binário não exibido.
01-Audio/audio_app.png → 01-Audio/Images/audio_app.png
Ver Arquivo
Lado a Lado Deslizar Sobrepor

Antes

Largura:  |  Altura:  |  Tamanho: 853 KiB

Depois

Largura:  |  Altura:  |  Tamanho: 853 KiB

01-Audio/README.md
+1 -1
Ver Arquivo
@@ -1,7 +1,7 @@
# Speech Emotion Recognition # Speech Emotion Recognition
![image](audio_app.png) ![image](Images/audio_app.png)
The aim of this section is to explore speech emotion recognition techniques from an audio recording. The aim of this section is to explore speech emotion recognition techniques from an audio recording.
02-Text/README.md
+4 -4
Ver Arquivo
@@ -1,6 +1,6 @@
# Text-based Personality Traits Recognition # Text-based Personality Traits Recognition
![image](text_app.png) ![image](/00-Presentation/Images/text_app.png)
In this section you will find all resources, models and Python scripts relative to text-based personality traits recognition. In this section you will find all resources, models and Python scripts relative to text-based personality traits recognition.
@@ -47,7 +47,7 @@ Gensim : 3.4.0
## Pipeline ## Pipeline
![image](/Presentation/Images/text_pipeline.png) ![image](/00-Presentation/Images/text_pipeline.png)
The text-based personality recognition pipeline has the following structure : The text-based personality recognition pipeline has the following structure :
- Text data retrieving - Text data retrieving
@@ -66,7 +66,7 @@ The text-based personality recognition pipeline has the following structure :
## Model ## Model
We have chosen a neural network architecture based on both one-dimensional convolutional neural networks and recurrent neural networks. We have chosen a neural network architecture based on both one-dimensional convolutional neural networks and recurrent neural networks.
The one-dimensional convolution layer plays a role comparable to feature extraction : it allows finding patterns in text data. The Long-Short Term Memory cell is then used in order to leverage on the sequential nature of natural language : unlike regular neural network where inputs are assumed to be independent of each other, these architectures progressively accumulate and capture information through the sequences. LSTMs have the property of selectively remembering patterns for long durations of time. The one-dimensional convolution layer plays a role comparable to feature extraction : it allows finding patterns in text data. The Long-Short Term Memory cell is then used in order to leverage on the sequential nature of natural language : unlike regular neural network where inputs are assumed to be independent of each other, these architectures progressively accumulate and capture information through the sequences. LSTMs have the property of selectively remembering patterns for long durations of time.
Our final model first includes 3 consecutive blocks consisting of the following four layers : one-dimensional convolution layer - max pooling - spatial dropout - batch normalization. The numbers of convolution filters are respectively 128, 256 and 512 for each block, kernel size is 8, max pooling size is 2 and dropout rate is 0.3. Our final model first includes 3 consecutive blocks consisting of the following four layers : one-dimensional convolution layer - max pooling - spatial dropout - batch normalization. The numbers of convolution filters are respectively 128, 256 and 512 for each block, kernel size is 8, max pooling size is 2 and dropout rate is 0.3.
Following the three blocks, we chose to stack 3 LSTM cells with 180 outputs each. Finally, a fully connected layer of 128 nodes is added before the last classification layer. Following the three blocks, we chose to stack 3 LSTM cells with 180 outputs each. Finally, a fully connected layer of 128 nodes is added before the last classification layer.
@@ -74,4 +74,4 @@ Following the three blocks, we chose to stack 3 LSTM cells with 180 outputs each
We tried different baseline models in order to assess the performance of our final architecture. Here are the accuracies of the different models. We tried different baseline models in order to assess the performance of our final architecture. Here are the accuracies of the different models.
![image](/Presentation/Images/perf_text_final.png) ![image](/00-Presentation/Images/perf_text_final.png)
02-Text/text_app.png
BIN
Ver Arquivo
Arquivo binário não exibido.
Lado a Lado

Antes

Largura:  |  Altura:  |  Tamanho: 840 KiB

04-WebApp/library/__pycache__/speech_emotion_recognition.cpython-35.pyc
BIN
Ver Arquivo
Arquivo binário não exibido.
04-WebApp/library/__pycache__/text_emotion_recognition.cpython-35.pyc
BIN
Ver Arquivo
Arquivo binário não exibido.
04-WebApp/library/__pycache__/text_preprocessor.cpython-35.pyc
BIN
Ver Arquivo
Arquivo binário não exibido.
04-WebApp/library/__pycache__/video_emotion_recognition.cpython-35.pyc
BIN
Ver Arquivo
Arquivo binário não exibido.
04-WebApp/static/js/audio_emotions_dist_other.txt
+8
Ver Arquivo
@@ -0,0 +1,8 @@
EMOTION,VALUE
Angry,9
Disgust,19
Fear,2
Happy,2
Neutral,43
Sad,22
Surprise,0
04-WebApp/static/js/db/audio_emotions.txt
+15
Ver Arquivo
@@ -1,5 +1,6 @@
EMOTIONS EMOTIONS
Neutral Neutral
<<<<<<< HEAD
Neutral Neutral
Neutral Neutral
Neutral Neutral
@@ -11,3 +12,17 @@ Disgust
Disgust Disgust
Sad Sad
Neutral Neutral
=======
Disgust
Disgust
Disgust
Angry
Angry
Angry
Disgust
Disgust
Disgust
Angry
Angry
Angry
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/audio_emotions_dist.txt
+7 -2
Ver Arquivo
@@ -1,8 +1,13 @@
EMOTION,VALUE EMOTION,VALUE
<<<<<<< HEAD
Angry,0 Angry,0
Disgust,16 Disgust,16
=======
Angry,46
Disgust,46
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
Fear,0 Fear,0
Happy,0 Happy,0
Neutral,75 Neutral,7
Sad,8 Sad,0
Surprise,0 Surprise,0
04-WebApp/static/js/db/audio_emotions_other.txt
+29
Ver Arquivo
@@ -597,6 +597,7 @@ Neutral
Neutral Neutral
Neutral Neutral
Neutral Neutral
<<<<<<< HEAD
Angry Angry
Angry Angry
Angry Angry
@@ -621,6 +622,8 @@ Angry
Disgust Disgust
Angry Angry
Angry Angry
=======
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
Neutral Neutral
Neutral Neutral
Neutral Neutral
@@ -629,9 +632,19 @@ Neutral
Neutral Neutral
Neutral Neutral
Neutral Neutral
<<<<<<< HEAD
Disgust Disgust
Disgust Disgust
Sad Sad
=======
Neutral
Neutral
Neutral
Neutral
Neutral
Neutral
Neutral
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
Neutral Neutral
Neutral Neutral
Neutral Neutral
@@ -643,5 +656,21 @@ Neutral
Neutral Neutral
Disgust Disgust
Disgust Disgust
<<<<<<< HEAD
Sad Sad
Neutral Neutral
=======
Neutral
Disgust
Disgust
Disgust
Angry
Angry
Angry
Disgust
Disgust
Disgust
Angry
Angry
Angry
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/hist_vid_glob.txt
+10
Ver Arquivo
@@ -1,4 +1,5 @@
EMOTION,VALUE EMOTION,VALUE
<<<<<<< HEAD
Angry,18 Angry,18
Disgust,8 Disgust,8
Fear,12 Fear,12
@@ -6,3 +7,12 @@ Happy,235
Sad,323 Sad,323
Surprise,44 Surprise,44
Neutral,436 Neutral,436
=======
Fear,6
Disgust,5
Angry,17
Sad,71
Neutral,208
Surprise,17
Happy,103
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/hist_vid_perso.txt
+10
Ver Arquivo
@@ -1,4 +1,5 @@
EMOTION,VALUE EMOTION,VALUE
<<<<<<< HEAD
Angry,0 Angry,0
Disgust,3 Disgust,3
Fear,0 Fear,0
@@ -6,3 +7,12 @@ Happy,24
Sad,0 Sad,0
Surprise,0 Surprise,0
Neutral,8 Neutral,8
=======
Fear,2
Disgust,5
Angry,4
Sad,21
Neutral,32
Surprise,5
Happy,18
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/text.txt
+5 -1
Ver Arquivo
@@ -151,8 +151,8 @@ Agreeableness,Conscientiousness,Extraversion,Neuroticism,Openness
0.2909279465675354,0.3018627464771271,0.2157977819442749,0.18610794842243195,0.00530355516821146 0.2909279465675354,0.3018627464771271,0.2157977819442749,0.18610794842243195,0.00530355516821146
0.1821022480726242,0.15624618530273438,0.1497754454612732,0.3081181347370148,0.20375804603099826 0.1821022480726242,0.15624618530273438,0.1497754454612732,0.3081181347370148,0.20375804603099826
0.1821022480726242,0.15624618530273438,0.1497754454612732,0.3081181347370148,0.20375804603099826 0.1821022480726242,0.15624618530273438,0.1497754454612732,0.3081181347370148,0.20375804603099826
0.2476610541343689,0.22067716717720032,0.231057733297348,0.01261020451784134,0.2879939079284668
0.2476610541343689,0.22067716717720032,0.23105773329734802,0.01261020451784134,0.2879939079284668 0.2476610541343689,0.22067716717720032,0.23105773329734802,0.01261020451784134,0.2879939079284668
0.2476610541343689,0.22067716717720032,0.231057733297348,0.01261020451784134,0.2879939079284668
0.2447648197412491,0.2237109392881393,0.2280386835336685,0.010952294804155828,0.292533278465271 0.2447648197412491,0.2237109392881393,0.2280386835336685,0.010952294804155828,0.292533278465271
0.2447648197412491,0.2237109392881393,0.2280386835336685,0.010952294804155828,0.292533278465271 0.2447648197412491,0.2237109392881393,0.2280386835336685,0.010952294804155828,0.292533278465271
0.0835515558719635,0.03762182593345642,0.11377312988042833,0.4335751831531525,0.3314782679080963 0.0835515558719635,0.03762182593345642,0.11377312988042833,0.4335751831531525,0.3314782679080963
@@ -209,9 +209,13 @@ Agreeableness,Conscientiousness,Extraversion,Neuroticism,Openness
0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306 0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306
0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306 0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306
0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306 0.2451882064342499,0.22297443449497226,0.22819821536540985,0.010884138755500315,0.29275497794151306
<<<<<<< HEAD
0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.3102996647357941,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
0.31029966473579407,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827 0.31029966473579407,0.18111537396907806,0.296110063791275,0.04382285103201866,0.1686520278453827
=======
0.24524909257888794,0.2229149490594864,0.22827190160751343,0.010896085761487484,0.292667955160141
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/text_mean.txt
+8
Ver Arquivo
@@ -1,6 +1,14 @@
Trait,Value Trait,Value
<<<<<<< HEAD
Extraversion,0.21900552004161808 Extraversion,0.21900552004161808
Neuroticism,0.08546119306184766 Neuroticism,0.08546119306184766
Agreeableness,0.2293632908689755 Agreeableness,0.2293632908689755
Conscientiousness,0.19947181524346685 Conscientiousness,0.19947181524346685
Openness,0.26669816076927993 Openness,0.26669816076927993
=======
Conscientiousness,0.20010490425513677
Extraversion,0.21685689027416763
Agreeableness,0.22713707076712242
Openness,0.2696092820580678
Neuroticism,0.0862918326110354
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/text_perso.txt
+8
Ver Arquivo
@@ -1,6 +1,14 @@
Trait,Value Trait,Value
<<<<<<< HEAD
Extraversion,0.296110063791275 Extraversion,0.296110063791275
Neuroticism,0.04382285103201866 Neuroticism,0.04382285103201866
Agreeableness,0.31029966473579407 Agreeableness,0.31029966473579407
Conscientiousness,0.18111537396907806 Conscientiousness,0.18111537396907806
Openness,0.1686520278453827 Openness,0.1686520278453827
=======
Conscientiousness,0.2229149490594864
Extraversion,0.22827190160751343
Agreeableness,0.24524909257888794
Openness,0.292667955160141
Neuroticism,0.010896085761487484
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
04-WebApp/static/js/db/words_common.txt
+12 -6
Ver Arquivo
@@ -119,6 +119,7 @@ computer,14
computing,2 computing,2
conception,4 conception,4
concise,2 concise,2
confident,1
confront,2 confront,2
consensus,4 consensus,4
consider,16 consider,16
@@ -195,7 +196,7 @@ e,8
earlier,4 earlier,4
early,2 early,2
economic,2 economic,2
ecosystem,4 ecosystem,5
edge,2 edge,2
effort,2 effort,2
email,4 email,4
@@ -227,7 +228,7 @@ facial,4
far,2 far,2
fatality,2 fatality,2
favourable,4 favourable,4
feel,6 feel,7
field,10 field,10
finance,6 finance,6
financial,16 financial,16
@@ -294,7 +295,7 @@ hundred,4
hybrid,2 hybrid,2
idea,4 idea,4
illustrate,4 illustrate,4
impact,4 impact,5
important,6 important,6
importantly,2 importantly,2
impress,4 impress,4
@@ -366,7 +367,7 @@ lettrepe,6
level,2 level,2
leverage,2 leverage,2
life,2 life,2
like,10 like,11
lionel,4 lionel,4
list,2 list,2
london,10 london,10
@@ -468,6 +469,7 @@ pleasure,4
point,4 point,4
portfolio,4 portfolio,4
position,10 position,10
positive,1
post,2 post,2
precise,4 precise,4
preferred,2 preferred,2
@@ -501,7 +503,7 @@ rank,8
raphael,6 raphael,6
reach,2 reach,2
read,4 read,4
ready,2 ready,3
real,4 real,4
realise,4 realise,4
realize,2 realize,2
@@ -560,7 +562,7 @@ sit,2
situation,2 situation,2
six,4 six,4
sixth,2 sixth,2
skill,12 skill,13
societe,4 societe,4
sound,2 sound,2
space,2 space,2
@@ -649,7 +651,11 @@ welcome,2
well,8 well,8
willing,4 willing,4
word,2 word,2
<<<<<<< HEAD
work,38 work,38
=======
work,33
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
working,2 working,2
would,18 would,18
write,4 write,4
04-WebApp/static/js/db/words_perso.txt
+12
Ver Arquivo
@@ -1,4 +1,5 @@
WORDS,FREQ WORDS,FREQ
<<<<<<< HEAD
work,1 work,1
group,1 group,1
project,3 project,3
@@ -25,3 +26,14 @@ top,1
downloaded,1 downloaded,1
apps,1 apps,1
lately,1 lately,1
=======
feel,1
like,1
confident,1
impact,1
ready,1
work,1
skill,1
positive,1
ecosystem,1
>>>>>>> f5d77ca6bf9ee142e8625d0da5f5140497ddd45d
README.md
+3 -3
Ver Arquivo
@@ -6,7 +6,7 @@ Don't hesitate to ⭐ the repo if you enjoy our work !
## In a nutshell ## In a nutshell
We developped a multimodal emotion recognition platform to analyze the emotions of job candidates, in partnership with the French Employment Agency. We developped a multimodal emotion recognition platform to analyze the emotions of job candidates, in partnership with the French Employment Agency.
We analye facial, vocal and textual emotions, using mostly deep learning based approaches. We deployed a web app using Flask : We analye facial, vocal and textual emotions, using mostly deep learning based approaches. We deployed a web app using Flask :
@@ -117,7 +117,7 @@ To limit overfitting, we tuned the model with :
- And kept the best model - And kept the best model
<p align="center"> <p align="center">
<img src="/Presentation/Images/Accuracy_Speech.png" width="400" height="400" /> <img src="/00-Presentation/Images/Accuracy_Speech.png" width="400" height="400" />
</p> </p>
### c. [Video Analysis](https://github.com/maelfabien/Multimodal-Emotion-Recognition/tree/master/Video) ### c. [Video Analysis](https://github.com/maelfabien/Multimodal-Emotion-Recognition/tree/master/Video)
@@ -199,5 +199,5 @@ https://www.overleaf.com/read/xvtrrfpvzwhf
<td align="center"><a href="https://github.com/STF-R"><img src="https://avatars0.githubusercontent.com/u/43505879?v=4" width="100px;" alt="mfix22"/> <td align="center"><a href="https://github.com/STF-R"><img src="https://avatars0.githubusercontent.com/u/43505879?v=4" width="100px;" alt="mfix22"/>
<br /><sub><b>STF-R</b></sub> <br /><sub><b>STF-R</b></sub>
</a><br /><a href="https://github.com/maelfabien/Multimodal-Emotion-Recognition/commits?author=STF-R" title="Code">💻</a></td> </a><br /><a href="https://github.com/maelfabien/Multimodal-Emotion-Recognition/commits?author=STF-R" title="Code">💻</a></td>
</tr></table> </tr></table>