Converted from a Word document
With the explosion of digitized textual legal resources and advances in computational natural language processing, scholars have begun application of automated content analysis to the study of legal texts (Livermore / Rockmore 2019). Applying computational methods to the reading and analysis of legal texts, namely treating textual narratives as data (Grimmer / Stewart 2013; Nguyen et al. 2019) affects the way scholars read these texts. Performing “distant reading” (Moretti 2000, 2013) scholars read texts based on analysis of statistics, when the “reading” of the text is actually performed by the machine using a series of algorithms. Distant reading approach to narrative documents holds the potential to expand our understanding beyond the human and readerly scale which is inherently constrained by scale and remains limited to a small selection of texts. This approach however risks losing sight of the narrativity that defines testimonial documents that is often extracted and imbued with meaning through a practice of “close reading”. Borrowed from literary criticism, close reading refers to the careful, sustained interpretation of a brief passage of a text. In law, a field defined by textuality, the analytical engagement with texts and extracting meaning from them based on a close attention to individual words, syntax, order of sentences and other formal structures, is often used to interpret legal texts such as judicial opinions, laws and regulations, constitutions and more.
In this paper we examine an untapped source of testimonial narratives found in the protocols of the Israeli Or State Commission of Inquiry that investigated lethal clashes between Israeli police and Arab minority in October 2000, resulting in the killing of 13 Arab-Israeli citizens shot by police forces.
With more than 2,000,000 words, the Or corpus exceeds a readerly scale. On the other hand, a process of selection of a handful of testimonies risks losing sight of “the bigger picture” and remaining tied to a certain limited perspective – that of a specific officer or a specific witness, which is ultimately anecdotal. What is missing is a dynamic reading model for analyzing the entire data from the Or Commission in an aggregated manner, while also zooming in on specific narratives of legitimation. In legal scholarship, few attempts have been made thus far to bridge the gap between close and distant reading (Keydar 2020). To gauge this gap, we develop a model of scaled reading which synthesizes familiar humanistic approaches of close reading with computational distant reading (Long / So 2016). The proposed scaled reading model moves between three scales of analysis: large-scale at the corpus level, mid-scale at the topic level and small-scale at the document level.
The aim of the first level was to identify the main themes included in the corpus. At the basis of the model, we employ a statistical process of unsupervised topic modeling (Blei et al. 2003). Topic modeling is an exploratory technique, useful for imposing order upon large bodies of textual data. The model captures the latent semantic structure behind a text collection. It allows for a compressed representation of text in a low dimensional space of comprehensible topics that contextualize word meaning. As such, topic models are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable (Nguyen et al. 2019). Unlike qualitative analysis based on information retrieval, where researchers know what they are looking for, topic models are attractive because they offer a formalism for exposing a corpus’ themes by discovering groups of words that often appear together in documents, given the namesake “topics” (Blei 2012). By a topic being “about” a theme, we mean that those distributions over the vocabulary of the corpus place high probability on words that a domain expert would interpret as related to the subject (DiMaggio et al. 2013). Unlike other empirical methods, topic models do not require the translation or reduction of documents into binary or numerical form (Law 2016), making them especially attractive to the analysis of complex narratives, such as testimonies, as well as other free-form legal texts.
In the paper we use a topic model that builds on latent Dirichlet allocation (
LDA
) (Blei et al. 2003), which has become the industry standard, also gaining
traction in computational legal analysis (DiMaggio et al. 2013; Livermore /
Rockmore 2017; Ruhl et al. 2018). LDA is a
hierarchical probabilistic model that represents each topic as a distribution
over terms and represents each document as a mixture of the topics“ (Dieng et
al. 2020). When fit to a collection of documents, – in our case the Or protocols
and report – the topics summarize their contents, and the topic proportions
provide a low-dimensional representation of each document (Dieng et al. 2020).
We set the number of topics on K=15, based on the relative size of the corpus,
repeated occurrence of key topics and the level of interpretability, which were
tested on different K values from 5 to 20.
The aim of the second level is to identify and categorize the voices that participate in the corpus and connect between categories of speakers and the themes arising from the topical analysis. The model scales down from the corpus level to the topic level, detecting – for each topic – the speakers’ group which is most associated with that topic. By connecting between identities and topics in the corpus, we can detect which themes and arguments were taken up by each of the groups, or even originated with them. As part of the mid-scale reading, we also considered the files of the Or Commission’s final report as representative of a specific speaker group, that of the State-appointed investigative body. Unlike the individual speech acts by the Commissioners during the proceedings (which were separately labeled as “judges”), the report represents a unified perspective of the official Commission of Inquiry, labeled separately as “report.”
The aim of the final level was to gain understanding of the context in which topics arise, and the rhetoric underlying each topic. Our proposed model of scaled reading allows the reader to focus the analytical attention on a small set of
computationally identified texts from the entire corpus. We developed a statistical model that allows us to identify the documents that best represents each topic. We were interested in selecting the documents that are not only those with the highest
topic probability for any given topic (i.e. the document that includes highest “concentration” of the topic), but rather—because this could be an outlier of either the text or the algorithm—the document that
best represents how the topic regularly “behaves” in relation to all other topics within the corpus.
We conceptualized
best representing document as being of highest topic probability and typicality to the set of documents relevant to the topic. We defined the threshold for
highest probability using a percentile-based cut off, that takes into account the changing range of probability distribution within each topic.
High typicality takes into account the relation between the coinciding topics that appear together with the discussed topic in the set. Measured as the15-dimensional distance from the centroid of the set. Each selection creates a different set of ranked documents by distance from the centroid. We selected the parameters that were least sensitive to changes in the threshold values and distance measuring, choosing percentile 30% and Euclidean distance.
Using this process of selection for identifying the most representative documents for each topic, the small-scale reading anchors the choice of which texts to examine individually and interpret as reflective of the entire topic, which is a main challenge when dealing with a large
narratival corpus. The small-scale reading method provides a more robust basis for the selection process, diminishing the risk of anecdotal evidence. As one scholar noted, “Computational methods can help us with the empiricism that is crucial to accurate close reading.” (Eve 2019: 349).
Moving back and forth between the computational topic modeling-based large-scale analysis and the mathematically assisted small-scale qualitative analysis, the three-tiered scaled reading model overcomes the distant / close reading dichotomy which dominates contemporary scholarship in computational text analysis. The scaled reading model allows us to combine the strengths of each of the methods, so that we can identify large-scale patterns in the corpus without losing sight of individual voices and stories.