Multi-format publishing and repurposing of historical linguistics data

1. Introduction

This paper stems from a project that studies modality in Latin from a diachronic perspective (WoPoss 2019).¹ We present the outputs produced by the project and how they are tailored to different purposes. Our strategy can be easily implemented by other projects dealing with similar issues: heterogeneous sources and multiple layers of annotation that require different workflows (e.g., the combination of automatic and manual annotation).

In recent years, much focus was put into the definition of data management practices in the scientific context. A significant milestone in the creation of a common framework for the construction and administration of scientific data was reached with the publication of the FAIR principles (Wilkinson et al. 2016). In addition to these cross-disciplinary guidelines, specific recommendations that cover the particularities of Humanities research have been presented (Harrower et al. 2020). Thus, complying with the principles of open science, our project has been implementing data sharing policies from its early stages (Dell’Oro et al. 2020).

We will first present the different phases in which our data processing is articulated. We will then focus on how the publication of our materials in multiple formats increases its reusability and interoperability.

2. An open science workflow

Our data collection begins by gathering an initial dataset from different open, online resources. Besides taking into consideration the diachronic perspective, we are interested in semantic change that may have emerged in any of the Latin varieties. This means that our corpus needs to be representative in terms of chronology, provenance, source typology and textual genre, among other sociolinguistic factors. This requirement compels us to collect texts from heterogeneous resources that provide their data in different formats. The collected texts are converted into plain text, but pseudo-markup is added to preserve relevant information that was previously described in the markup.

These plain text files are then automatically annotated using Stanza (Qi et al. 2020). The results are output in CONLL-U format (Universal Dependencies contributors 2020). These files are then uploaded to the annotation platform INCEpTION (Klie et al. 2018). Using this platform, we annotate each modal passage according to our Guidelines (Dell’Oro 2019). A modal passage is identified on the basis of the presence of a predefined list of modal markers (eg. the verb possum or the gerundive suffix -ndum) (Fig. 1).

After the annotation is curated, the results are exported in the UIMA CAS XMI format; the advantages of this format to formalize linguistic annotations are discussed in Eckart de Castillo et al. (2017). UIMA provides a standard for annotating unstructured data that enables the interaction between multiple layers of annotation, and CAS is the data structure through which all UIMA components communicate.

Example of modal passages annotated in INCEpTION

Although UIMA CAS XMI is a format used in the NLP community, its use is not as widespread among Humanities researchers as other formats. Thus, to increase the reusability of our dataset, we perform a post-processing that transforms this format into TEI. Our initial pseudo-markup also gets formalized as TEI elements at this stage. Metadata about the work (including information about the author) is also included, together with philological information provided by our annotators during the annotation process (in particular, in the case the used text deviates from the edition of reference we have chosen).

Together with the annotation, we also make a detailed study of the modal markers focusing on their semantic change. The purpose of this study is twofold: First, enriching the annotation by adding the most ancient modal meaning of each marker. Second, creating diachronic interactive visualizations of the semantics of each marker (Bermúdez-Sabel et al. 2020). These maps will be enhanced with the results of our empirical corpus-based analysis, once the annotation will be completed.

In addition, we will create an ontology, compliant with the Linking Latin (LiLa) model (Passarotti et al. 2020). Our goal is to convert our annotations to an RDF serialization that can be integrated in the LiLa Knowledge Base.

The complete annotated dataset will be accessible through a user-friendly interface (GUI) with faceted search, which will exhibit and exploit the fine-grained annotation. All the code and the resources that are used during development are open-source (WoPoss 2020). Our own programs are made openly available, and all external software the project relies on is also open source.

3. Multi-format publishing as a reusability strategy

The use of open standards is key for data sharing (Harrower et al. 2020: 19–20). We publish our data in different formats so that they can be repurposed according to the needs of different communities:

• Plain text version of the source texts. As the input format of many computational resources, this can be helpful also for literary and historical research.
• Diachronic modal semantic maps in JSON, SVG, and PNG (WoPoss 2021). They are useful for researchers of any field working with natural language, like history or philology, and they are also appropriate for conveying complex semantic concepts to students, thanks to the readability of the visual representation.
• Linguistically annotated materials (lemmata, part of speech and morphological features) in CONLL-U format. These materials can be used to add other layers of annotation.
• Semantically annotated modal passages in:
  • UIMA CAS XMI, intended for the NLP community.
  • TEI, to promote usability among the Humanities community.
  • RDF/XML: the publication as Linked Open Data increases interoperability besides bringing to the corpus the advantages of the Semantic Web ecosystem.
• Data access through a GUI to serve both a scientific and a non-specialized community.
• Web application. The dataset and the functionalities of the interface will be published as an eXist-DB application (Siegel / Retter 2014) so anybody can locally install it and, for example, increase the dataset with their own materials.

4. Conclusion

In this paper we discussed the advantages of publishing in different formats, at every step of development, the data constructed in the context of a historical linguistics project. We argue that this strategy increases the possibility of repurposing data, thus becoming a more useful asset to the scientific community.