TY - GEN
T1 - Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment
AU - Matero, Matthew
AU - Hung, Albert
AU - Schwartz, H. Andrew
N1 - Publisher Copyright: © 2022 Association for Computational Linguistics.
PY - 2022
Y1 - 2022
N2 - Recent works have demonstrated ability to assess aspects of mental health from personal discourse. At the same time, pre-trained contextual word embedding models have grown to dominate much of NLP but little is known empirically on how to best apply them for mental health assessment. Using degree of depression as a case study, we do an empirical analysis on which off-the-shelf language model, individual layers, and combinations of layers seem most promising when applied to human-level NLP tasks. Notably, we find RoBERTa most effective and, despite the standard in past work suggesting the second-to-last or concatenation of the last 4 layers, we find layer 19 (sixth-to last) is at least as good as layer 23 when using 1 layer. Further, when using multiple layers, distributing them across the second half (i.e. Layers 12+), rather than last 4, of the 24 layers yielded the most accurate results.
AB - Recent works have demonstrated ability to assess aspects of mental health from personal discourse. At the same time, pre-trained contextual word embedding models have grown to dominate much of NLP but little is known empirically on how to best apply them for mental health assessment. Using degree of depression as a case study, we do an empirical analysis on which off-the-shelf language model, individual layers, and combinations of layers seem most promising when applied to human-level NLP tasks. Notably, we find RoBERTa most effective and, despite the standard in past work suggesting the second-to-last or concatenation of the last 4 layers, we find layer 19 (sixth-to last) is at least as good as layer 23 when using 1 layer. Further, when using multiple layers, distributing them across the second half (i.e. Layers 12+), rather than last 4, of the 24 layers yielded the most accurate results.
UR - https://www.scopus.com/pages/publications/85137678613
M3 - Conference contribution
T3 - WASSA 2022 - 12th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Proceedings of the Workshop
SP - 89
EP - 94
BT - WASSA 2022 - 12th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Proceedings of the Workshop
A2 - Barnes, Jeremy
A2 - De Clercq, Orphee
A2 - Barriere, Valentin
A2 - Tafreshi, Shabnam
A2 - Alqahtani, Sawsan
A2 - Sedoc, Joao
A2 - Klinger, Roman
A2 - Balahur, Alexandra
PB - Association for Computational Linguistics (ACL)
T2 - 12th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA 2022
Y2 - 26 May 2022
ER -