Improving multilingual lexicon-based sentiment analysis for Scandinavian languages

1. Introduction

In this paper, we address the challenge of multilingual sentiment analysis using traditional lexicon and rule-based tools, since these tools are inherently tailored to capture sentiment patterns in a particular language. Focusing on a case study of three closely related Scandinavian languages (Danish, Norwegian, and Swedish) using three tailored versions of VADER, we measured the relative degree of variation in valence using the OpenSubtitles corpus. We found that scores for Swedish systematically skewed lower their translational pairs in Danish, and that scores for Norwegian skewed higher than for both other languages. We then trained a neural network to optimize the fit of Swedish > Danish and Norwegian > Danish, effectively correcting this skew.

2. Background

Lexicon-based tools are widely used in automated sentiment analysis and opinion-mining tasks, and remain the most accessible and easy to validate method of performing sentiment analysis on a text corpus. One of the most popular tools of this kind is Valence Aware Dictionary and sEntiment Reasoner (VADER) (Hutto / Gilbert 2014).

Prior to this study, cross-linguistic comparative sentiment analysis using VADER required the use of the multilingual version of VADER (https://pypi.org/project/vader-multi/), which uses integration with Google Translate API to automatically translate the source language into English to produce sentiment scores. However, translation-based approaches are inadequate for high-fidelity multilingual sentiment analysis given the high degree of variation in the valency in the translational equivalents of evaluative, subjective, and expressive words and expressions (Jackson et al. 2019; Mohammed et al. 2016).

To avoid the problem of translation, we use an alternative approach to cross-linguistic VADER sentiment analysis which applies a normalization algorithm, trained on a parallel corpus, to adjust language-specific VADER model scores for Danish, Norwegian, and Swedish texts making them more comparable.

3. VADER Sentiment Analysis tools in Danish, Swedish, and Norwegian

The sentiment dictionaries used in this study build upon VADER (Hutto / Gilbert 2014), a dictionary- and rules-based approach to sentiment analysis on social media. The new Danish VADER builds upon previous rule-based approaches to Danish sentiment analysis namely SENTIDA (Lauridsen et al. 2019), EMMA (Kran / Orm 2020), and AFINN (Nielsen 2011) and extends upon them by adding support emoji and replacing the use of stems with lemmas. The Swedish utilized the publicly available VADER implementation of Swedish (Hutto 2017) while the Norwegian (Bokmål) used a translation of the Danish corpus translated and validated by two native speakers. As the dictionaries used are fairly small¹ all models utilize lemmatization using Stanza (Peng et al. 2020) as opposed to the full word as the English² VADER to obtain much wider coverage. These versions are freely available online on https://github.com/centre-for-humanities-computing/text to x.

Note that comparatively better performance have been found for sentiment analysis using attention-based model such as BERT (Devlin et al. 2019), but these treat sentiment as a classification problem (for instance, GLUE reduces the sst-2 to a classification task (Wang et al. 2019)).³

Table 1: Total number of OpenSubtitles parallel sentences and mean VADER compound score

4. Parallel translation corpus for sentiment tool validation

The next step was to conduct pairwise comparisons of Danish, Swedish and Norwegian VADER sentiment tools using a corpus of translated movie subtitles (OpenSubtitles v.2018). From this corpus we sampled 1.9 million parallel sentence pairs for each language pair. We then applied our sentiment tools and calculated the average score (see Table 1).

Figure 1: Original VADER scores in Norwegian (blue), Swedish (yellow) and Danish (red) against each other. (Data: whole dataset reported in Table 1.)

From Table 1, we can see that in comparison to Danish VADER, our Norwegian VADER tool has a tendency to assign higher sentiment scores, while the Swedish VADER tool – lower scores. From Figure 1 (centre), we can also see that Norwegian and Swedish VADER tools are similarly aligned in relation to the Danish version. We therefore selected Danish VADER as the baseline, and developed Swedish-to-Danish and Norwegian-to-Danish models to improve alignment of the other two SA tools.

5. Model development and optimization

We first tried to identify data input features for optimal prediction of target Danish compound score. By testing various linear models and shallow artificial neural networks (Sequential model with 2-3 dense layers composed of 16, 32, 64 nodes) with different input features, we selected the following best features:

• VADER scores (neg, neu, pos, compound)
• one-hot encoded compound score range
  • extremely neg [-1...-0.4]
  • mild neg [-0.4...-0.2]
  • neu [-0.2...0.2],
  • mild pos [0.2...0.4]
  • extremely pos [0.4...1]

We applied Autokeras 1.0.2 [6] for automated neural network architecture selection with these features, and then optimized the hyperparameters using the sweep functionality from Weights and Biases [1] on a validation set consisting of 20% of the data. This was done using Bayesian grid search over the following parameters: number of units in each layer l1-5 (16-254), batch size (250-1000), dropout, and learning rate (0.0005-0.002). The resulting optimal parameters were: l1 111 units, l2 174 units, l3 225 units, l4 247 units, l5 36 units, batch size 623, dropout 0.06, learning rate 0.0005.

6. Effects of normalisation

A random sample (n=50,000) from the testing part of the dataset was selected for every language pair for evaluation of normalisation effects. As shown in the results in Table 2, normalisation produces almost identical mean sentiment values in parallel sentences, but has significantly decreased variation of sentiment scores in the normalised data.

Table 2: Adjusted mean VADER compound scores and standard deviation after normalisation. Non-normed scores in parentheses. (Data: the 50K sample from the OpenSubtitles test set for each language pair)

Furthermore, Table 3 shows that normalisation of Swedish and Norwegian sentiment scores has led to an increase in correlation (measured with non-parametric Spearman correlation test) and decrease in RMSE across all three language pairs.

Table 3: Spearman correlation r_s and RMSE before and after normalisation using the developed models. All reported correlation tests yielded p-value < 0.001. (Data: the 50K sample from the OpenSubtitles test set for each language pair).

7. Conclusion and future work

By applying a combination approach of language-specific sentiment scoring plus crosslingual score normalization, we hope to achieve a more satisfactory method for comparative lexicon-based sentiment analysis for these Scandinavian languages, without the use of automated translation.

Because this is work in progress, neither the baseline nor normalized scores produced by the three VADER models have yet been validated against a sentiment evaluation corpus. Existing resources (tagged sentiment corpora) for all three languages will be used in an upcoming benchmarking task, comparing our normalized scores with the compound scores of the three original VADER models and VADER multilingual.