Kusnick, Jakob
University of Southern Denmark, Denmark
kusnick@imada.sdu.dk
Focht, Josef
Musical Instrument Museum of Leipzig University, Germany
josef.focht@uni-leipzig.de
Jänicke, Stefan
University of Southern Denmark, Denmark
stjaenicke@imada.sdu.dk
Many historical musical instruments are resting in depots or are presented in showcases in music museums. Digitization is one of the current key strategies to gather data and to make those instruments virtually accessible through online repositories. But not all of them are linked together, and so relations between related objects remain hidden. Cooperating with musicologists, we unfold the history of musical instruments by “auralizing” (Summers 2008) them through matching compositions. The motivating research question is, in which particular performances of compositions a musical instrument was used. Therefore we assign musical pieces from RISM (1952), a digital collection of musical compositions, to musical instruments from MIMO (2011a), a digital repository of meta- and media-data on musical instruments in public collections. A multi-facetted similarity measure generates matching results between the two repositories automatically and interactive visualizations. We propose an extension of this process by a tag cloud picturing the aura of instruments of interest. The system is evaluated by musicologists who investigated different scenarios using the tool.
In our previous work (Kusnick et al. 2020), we proposed a system for the calculation and visualization of hypothetical relations between musical instruments from MIMO and musical pieces from RISM.
We take geospatial, temporal and descriptive meta-information into account to assign musical instruments to musical pieces. These attributes are used to define similarity measures which describe the likeness of a musical instrument playing a part in a particular piece performance. The result of the matching process is a list of musical instruments and the assigned musical piece performances. Schlegel et al. (2011) created a static visual timeline for types of lutes and lute-like instruments, enhanced with appropriate compositions. To visualize such pairings interactively, we designed a new timeline metaphor, where every instrument has its dedicated row. Distant and close reading research questions can be answered through different semantic zoom levels, what has previously been demonstrated for compositions (André et al. 2007). As described by Khulusi et al. (2020), data on musicology include uncertainty aspects, which are visually encoded in our approach. The timeline enables the explorative analysis of hypothesized matches and a map as an additional view helps the user to validate or falsify the automatically generated suggestions. So the instruments’ careers are visualized through their temporal and geographical located events in timeline and map. Now we propose an additional view to picture the aura of one or a group of musical instruments of interest.
The timeline and map generate visual patterns, and potentially interesting findings stick out. The previous version of the system only allowed to select an individual musical instrument after filtering the results via search terms and clicking a particular instrument, getting to a single close reading view. The aura of an individual or multiple instruments was just perceivable by a follow-on, laborious clicking on each matched piece performance. The extended system now enables to select instruments via a rectangular selection box by click-and-drag in the timeline or map. This selection filters the result set and leads to the refreshed multi-view screen of timeline and map, including a new visualization depicting the aura of instruments.
The aura of instruments is pictured in the form of a tag cloud. It shows the resulting instruments, described by words aggregated from the matched musical pieces. The arrangement of words is automatically determined by a force directed graph drawing algorithm (Kobourov 2012), in which each item is a node and the links between them represent joint appearances in the same category. This leads to forming clusters, inspired by TagPies (Jänicke et al. 2018). In the cloud’s center, each instrument is symbolized by its image, or a placeholder image in a circle. The words, whose sizes are determined according to the number of occurrences, are taken from four different attributes of musical works, and, additionally, the titles of the instruments. So the words appear in five different colors to quickly depict the instrument’s aura visually. In most cases, the largest cluster (in purple) is built by the tokenized titles of the musical pieces found in the matching process. Due to the variety of possible languages, we did not apply further language processing like stemming or lemmatization. Nevertheless, the token sets are cleaned through stopword lists of the most frequently used languages like German, English, and Italian.
Furthermore, the names of the musical piece composers constitute the second cluster (in green) of the tag cloud. Due to the length of full names, they were cropped and only the surname is displayed. Thanks to the additional meta-information about the composers, the user can inspect the details by hovering the name tags and see the composer’s full name and lifetime. Additionally, genre names (in red) and languages (in blue) of the manuscripts are displayed in separate word clusters of the tag cloud.
The visual exploration of a trombone (MIMO 2011b) originating Nuremberg, Germany has been described in our previous paper (Kusnick et al. 2020). The inspection of matched pieces on the map reveals that they were performed in some cities close to Nuremberg. The tag cloud highlights that the most frequently matching genre is the opera (closely related to “Singspiele”), and German (“ger”) as the major language. Frequent terms in the titles are “Zauberflöte” and “Giovanni”, in which trombones were used, as well as “Mozart” as the corresponding composer. A closer look reveals many terms related to religion. German cantatas, sacred songs and numerous names including words like “Herr” (lord), “herrlich” (wonderful), but also “Sünden” (sins) and “Opferfest” (sacrifice) from Peter von Winter and Georg Philipp Telemann are found in the resulting list of musical piece tags prominent in the word cloud. The musicologist working with this use case commented that the denomination was very important around 1800. Additionally, he took the predominant denomination of the piece performances’ locations into account. All cities were predominantly Protestant, except the Catholic Munich. Thus, the trombone from Nuremberg may be better auralized through Protestant songs from one of the culturally closer cities like Frankfurt am Main, even though it is geographically farther away than Munich. Clicking “Cantatas” shows a list of all cantatas in the musical piece matches, for example “Ertönet bald herrlich ihr letzten Posaunen” (Telemann 1741) (also tagged as a sacred song) having even “Posaune” (trombone) in its title. The song was performed in Frankfurt am Main and was composed by Georg Philipp Telemann who is tagged as evangelic in the musiXplora (musiXplora 2020).

Figure 1: The generated tag cloud for a trombone from Leipzig, Germany and its matched compositions within a range of 50 years and 200 kilometers.
The history of plucked instruments like lutes and lute-likes is pictured by a second use case. Three dedicated epochs are visible in the timeline in Figure 2, and the corresponding tag cloud is viewable in Figure 3a.
The first epoch (1620–1760) refers to the classical lute, indicating a very sparse occupancy due to the low amount of pieces dedicated for lutes in RISM. However, a wide distribution of genres is depicted in the word cloud in Figure 3b.
The second epoch (1770–1810) is characterized by numerous precise datings (see Figure 2) for the production of the instruments. This might be contractual works commissioned by the courtly society because lutes were highly appreciated instruments at that time. Also, the word cloud gets more dense and manifold (Figure 3c), especially, the amount of mandolins increases.
The third epoch (1870–1960) is marked by very uncertain production dates because of their more industrial production, visually encoded by more transparent and wider red ovals in the timeline. Also, this episode was the time where lutes and lute-like instruments only survived by being transformed into bass-lutes or guitars to play roles in operas because they were not part of uprising orchestras, as seen in Figure 3d.

Figure 2: All found lutes and lute-like instruments with their matching piece performances within a distance of 50 years and 100 kilometers around the instrument events. For details, see the original publication (Kusnick et al. 2020).

Figure 3: Four different tag clouds for lutes and lute-like instruments summarizing all pairings of instruments and compositions for the time ranges (a) 1620–1960, (b) 1620–1760, (c) 1760–1870, and (d) 1870–1960.
While historical musical instruments are resting in showcases or depots of museums, digitization efforts serve us with intangible data about them. This data could be used to make the musical instruments and their history perceptible. We extended our visualization system by an interactive tag cloud to picture the career and aura of an instrument or group of instruments of interest in a manifold way. The created aura is composed of word clusters for five different attributes of the matched composition or instrument giving an overview of the selected data subset. This visual depiction could be presented to visitors of museums next to showcases to contextualize instruments through their music and to bring their particular careers to life.