Converted from an OASIS Open Document
An important aspect of the analysis of poetry relies on the extraction of information from the different structures found in a poem. To a certain degree, identifying these structures automatically with the help of a computer is partially possible now and such tools already exist for many languages such as Spanish (Gervás 2000), Portuguese (Araújo / Mamede 2002), French (McAleese 2007), English (Heuser / Antiila 2010), Czech (Ibrahim / Plecháč 2011), Euskara (Agirrezabal 2016), or Dutch (De Sisto 2020). Most approaches use rule-based systems (Gervás 2000; McAleese 2007; Heuser / Antiila 2010; Ibrahim / Plecháč 2011; De la Rosa et al. 2020), while only a few make use of neural networks techniques (Agirrezabal 2016; De Sisto 2020). Remarkably, the automatic identification of stanza types remains understudied. In this work, we framed the problem of stanza type identification as a classification task.
A stanza is the minimal structural unit of a poem that usually also encapsulates themes or ideas (Kirszner 2003). Stanzas are formed by verses, and as such they are shaped by authorial styles and historic preferences. In the Spanish tradition, three aspects determine how a stanza is identified: metrical length, rhyme type, and rhyme scheme (Domínguez Caparrós 2014; Jauralde 2020; Quilis 2000; Torre 2000). Therefore, stanza classification can be formulated as a three stages process (Domínguez Caparrós 2014) (see Table 1 for an example):
Table 1. Example of a 6 verses stanza by poet Miguel Hernández with assonant rhyme showing metrical lengths, rhyme type, and rhyme scheme. Rhyming sound of the ending word in bold.
Once these three aspects are identified (verse lengths, rhyme types, and rhyme scheme), the expert has to match their characteristics against the different definitions of stanza types given by the poetic tradition and make a decision about the best match.
In order to test automatic approaches to stanza classification, we first collected a corpus of 5005 stanzas from 1600 poems ranging from the early 15th century to contemporary poems, containing a balanced number of stanzas for each of the 45 stanza types considered plus an extra category for stanzas with no type known. The corpus was manually reviewed by three experts to ensure texts were written in modern Spanish (computer tools tend to struggle with old Spanish) with no spelling or orthotypographic mistakes (Pérez Pozo et al., 2021). Based on the 3-stages of stanza classification aforementioned, we created a knowledge base with the appropriate rules for each type of stanza, and had the same group of experts decide on the order they should be applied and how ambiguities should be resolved. We then incorporated these rules to a baseline expert system built on top of Rantanplan (De la Rosa et al. 2020), a scansion tool for Spanish that reports state of the art results for metrical pattern extraction. Using the metrical information extracted by Rantanplan from the plain text of a stanza, and the set of manually crafted rules by the experts, our baseline was able to predict a stanza type with an accuracy of 78.63% when evaluated on the corpus. Once the baseline was established, we tested different techniques in an attempt to improve the classification results.
From a machine learning point of view, the problem is a pure multiclass classification task. Decision trees come as a natural fit to our problem for their capacity to encode rules and their priorities, and their ability to produce explainable and interpretable models. To carry out the training, we split the 5005 stanzas in our corpus into stratified training and testing sets of 4004 stanzas (80%) and 1001 stanzas (20%), respectively. Each stanza was vectorized and transformed into a feature vector containing information about what independent rules in the knowledge base were triggered according to the metrical information extracted by Rantanplan. That is, we ignored the experts’ knowledge on the order and subsumption of rules and gave the activations as inputs to the classifier. Tree-based methods were able to improve our baseline by almost 13% (see Table 2).
Table 2. Tree-based evaluation results expressed in accuracy percentages on the test set. Best result in bold.
We did not cross-validate the training and testing sets since the same splits were used to train computationally expensive neural networks. We expected that the networks would learn most of the rules necessary to classify the different stanza types without manually encoding the rules. Since embeddings have demonstrated to be highly effective in a number of NLP tasks (Chen et al. 2013), we relied on word embeddings and language models to extract the feature vectors from the plain text of each stanza in the corpus. The architecture was based on stacked bi-directional LSTM layers, dropout, and a fully connected linear layer acting as a classifier. The input to the neural networks was the plain text of each stanza. We then tested both GloVe (Pennington et al. 2014) and BERT (Devlin et al. 2019) text embeddings as inputs to the networks. After a small grid search, we found that GloVe performed better with 3 LSTM layers, dropout of 0.2, learning rate of 1.25e-5, and trained for 325 epochs. While BERT, with the same LSTM settings but a learning rate of 1e-4, plateau at 5 epochs. We also tested a hybrid approach in which we combined a BERT model with the vector of features extracted by Rantanplan (same vector used as input to the tree-based models). In this case, the model performed better with 20 LSTM layers, a dropout of 0.1, learning rate of 1e-3, and trained for 100 epochs. The models were trained using the same 80/20 split, but reserving 20% of the training set for evaluation purposes. Table 2 shows that this approach performed better than any other.
While notably better than a random classifier, our baseline expert system suffered from a series of limitations: the use of Old Spanish in some poems, which confused the underlying scansion tool; the relaxation of some rules related to verse length allowing a small fluctuation in the fixed length of verses (Domínguez Caparrós 2014; Jauralde 2020; Quilis 2000; Torre 2000); and the presence of hemistiches, verses split in two halves with independent metrical lengths that affect that of the verse as a whole. Tree-based solutions were capable of learning the right order in which rules needed to be applied and boost the accuracy by 13%. On the other hand, neither contextual (BERT) nor contextually-aware (GloVe) embeddings produced better results than our baseline. However, the combination of the contextual embeddings with prior domain-specific knowledge seems to perform over 15% better than our baseline. The embedding layers seem to carry insufficient structural information for this task, but it complements very well the feature set obtained from the 3-stage rules of each stanza type.
In this work, we have framed the problem of stanza identification as a classification task prone to be approached using machine learning techniques, including those based on recent natural language processing methods such as word embeddings and language models. In the process, we have contributed with a novel corpus of more than 5000 stanzas of 45 types in Spanish, a knowledge base with rules for each stanza type, and a baseline classifier soon to be released as part of an existing scansion tool. We have shown that powerful language models trained on billions of words are not yet capable to outperform simple methods such as decision trees, since the amount of structural information encoded in their dense vectors is not sufficient to perform stanza classification. Finally, we have proven that combining both expert knowledge and contextual embeddings performs best, producing a system that assigns a correct stanza type over 91% of the time. This result is both somewhat expected, since stanza classification is a very nuanced task, and challenging, since it might hinder our future plans to create models that perform well in several languages without explicitly encoding their rules. Moreover, a proper error analysis needs to be conducted to examine where the differences in performance are coming from.
All the source code and corpus are available at GitHub repository: https://github.com/linhd-postdata/stanza-detection-evaluation.
This work was supported by the Starting Grant research project ‘Poetry Standardization and Linked Open Data: POSTDATA (ERC-2015-STG-679528), funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program.