Growing together: evolving tools for evolving concepts

In the AdA-project the research process encompassed different steps such as creating annotation data (either computational or manually), exploring the data related to moving images, finding patterns and reconstructing figurations as well as communicating them to others. For the explorative comparative study of an extensive corpus, large data sets of detailed film-analytical, systematic, machine-readable annotations have been created. In this process, the Advene software has been involved at the various stages, from the exploration of data modelling and visualizations and the actual creation of annotation datasets, to the exploration of annotations in order to support analyses. Advene has been designed as a tool for audiovisual active reading (Aubert / Prié 2005; Aubert et al. 2012), allowing users to create and share annotation-based interpretations of audiovisual media. In order to become a part of the creative scholarly processes involved in the development of media analyses, it is built upon flexible data structures and visualizations. In the course of a scholarly activity, as underlined by Prié (2011), information structures are constantly evolving to various extents. Moreover, these evolving data structures are mediated by visualizations, that also have to evolve in order to match the needs of both structure evolution and thought process. The modalities and temporalities of these evolutions depend on the possibilities offered by the software used to mediate the information, and on the skills of the users: some evolutions of data or visualizations can consist in simple or more advanced configurations of existing tools or visualizations, that can be carried out autonomously up to a certain point. For more advanced evolutions of data structure and visualizations, more methodological and technological know-how may be involved and benefit from a collaboration with computer science scholars, who can provide insights and advice in modelling and configuration as well as possibly develop new tools when needed. The cooperation between computer science scholars and humanities scholars can neither be conceptualized as the ordering of a fixed product that fulfills pre-existing needs of humanities scholars, nor the deployment of state-of-the-art visualization modes by computer science scholars that are then just applied. In our experience, as can be also seen in El Kathib et al. (2019), interdisciplinary research in the realm of digital humanities rather requires joint processes of sketching out needs, possibilities and restrictions and translating affordances into the languages of both domains. Such translation requires inter-comprehension, which can be achieved through mutual appropriation of the domain-specific concepts, or at least have some members “able to speak across various disciplines”(Siemens et al. 2010). The ‘product’ of such a development process is thus more aptly described as a framework that enables scholars to think dynamically within a certain data-visualization paradigm.

Mapping time and movement: the development of a visualization framework for media studies

We ground these reflections on the evolution of timeline visualizations, that was used throughout the project, describing why and how it had to evolve in order to support the analyses of the humanities scholars, as well as the collaboration process that led to these evolutions. We will also underline some of the design principles that offer a level of empowerment to humanities scholars in adapting knowledge instruments. 

The challenge of creating visualizations of the temporal dynamics of audiovisual media has been approached in the realm of film studies from different angles and with diverging research interests: from the hand-drawn film scores by Russian formalists like Sergei Eisenstein (2010 [1941]) that focus on the synchronicity of various filmic modes to its digital reimagination (Heftberger 2018) and digital frameworks that focus on specific aesthetic dimensions like color (Flückiger / Halter 2020) or montage rhythm (Tsivian 2009). The main challenges in our case were the quantity of metadata, its semantic structure (Agt-Rickauer et al. 2018), the need for a systematized mode of visualization for a larger corpora through multiple annotators, while at the same time acknowledging the variance within the corpus. Even within a theoretical framework there is no one-fits-all-solution for visualizing the temporal dynamics of different films. Whereas in one sequence the cutting rhythm and character movements might be crucial, other sequences dominantly use music and changes in the color scheme to convey moods and meanings. Therefore the timeline visualizations are designed as sandboxes, that allow scholars without programming skills to rearrange, reshape and adjust the visual output on the fly within a restricted set of possibilities.

Adding configurability to a digital instrument actually increases its complexity from both the user’s and the designer’s point of view. The conception of a configurable instrument requires the creation of both the instrument itself, and also of its configuration means, which both define the possibilities it offers. In our case, the first version of the timeline (see Figure 1) was the one provided as a standard interface by Advene. It provides a generic way to interact with the annotations, both for creation and exploration. It offers different levels of configuration, which have been exploited during the project: the ability to define the presented types of annotations, its behavior when playing the associated video, etc. Some of the parameters like advanced color specification based on annotation content are accessible through a specific language syntax, that had to be appropriated by the humanities scholars in order to produce some variations in visualization, exhibiting characteristics of the data. However, the interaction facilities that the timeline offers can reduce its capacity to handle a larger number of annotations, and in the course of the project, the interdisciplinary discussions, exchanging requirements and constraints from both humanities and digital domains, led to the conclusion that another visualization component was needed. A new version of the timeline has thus been designed (see Figure 2 for a sketch used during the discussion phase, and Figure 3 for the new version), focusing more on data exploration than data creation and manipulation. One of the design features of the new version is the syntax used for describing the configuration of the timeline: it is a text-based (actually encoded as a human-readable URL) syntax, akin to a Domain-Specific Language (DSL) fitted for the visualization of the specific data structure of the AdA-project. This syntax presents a bit more complexity to use compared to a full-fledged graphical user interface, but it also offers more flexibility, allowing scholars to do independent variations and explorations with different settings. Moreover, its textual characteristic makes it more apt at being saved for further references or sharing, hence improving the reproducibility of the explorations. Additionally, not having to develop a full-fledged configuration interface, this approach allows to dedicate development time on core functionalities, in the constant tradeoff of finding a balance between the time gain offered by using a new feature and the time needed to develop it.

Fig. 1: Pre-existing timeline editing interface in Advene

Fig. 2:  Sketch used during the new timeline creation process

Fig. 3: New web-based version of the timeline for exploration and presentation

Reflections on an iterative & interdisciplinary process

These concrete procedures and considerations open up a discourse about the conceptualization of “software solutions” in the realm of digital humanities in a more general sense. Between the notions of ‘tool’ and ‘instrument’, the interdisciplinary process of the development appears as a manyfold and difficult process of crafting, appropriation, enablement and exploration. By reflecting on the instrumental genesis of a visualization framework, we aim in this paper at highlighting the process rather than the outcome. Especially in the realm of digital humanities these processes of inter- and transdisciplinary communication and exchange pose a major challenge for many research projects – especially when they work with challenging and complex data types. By mapping out the problems and possibilities that arise from the friction and multitude of perspectives not as an abstract state but as organic parts of a concrete development process, we try to bridge the gap between experience report and theoretical self-reflection