Measuring Communication Quality of Interest Rate Announcements

3.1 Text Accessibility

We measure the accessibility of text within the interest rate announcements using the following two indices. The first one is the type-token ratio (TTR), which captures how the vocabulary used is varied and is calculated as the number of different words (e.g. types) in the text compared to the number of times these words appear (e.g. the number of tokens per type):

A high TTR value indicates the use of a broad vocabulary within the text that requires more significant effort and knowledge to understand.

The second index, text complexity, estimates the number of years of U.S. grade-school-level education required to understand a given text (Kincaid et al. 1975; Flesch 1979). Since long words and sentences increase communication complexity, the index compares the number of words and sentences with the number of syllables and words,^[4] as follows:

The initial formula at the origins of Eq. (2), upon which Kincaid et al. (1975) based their results, was developed by Flesch (1948) in an experiment conducted on school-aged children. Using McCall-Crabbs’ standard test lessons in reading, Flesch (1948) derived from these experiments correlations between word length (wl), sentence length (sl), and the average grade of children who could answer at least 50 percent of the test questions correctly.^[5] These regressions led to the proposed Reading Ease (RE) formula: RE = 206.835 − 0.846wl − 1.015sl. This formula was tested by Flesch (1948) to match various texts and RE results, showing that the correlation between RE and effective RE, measured as the ability of children to understand the suggested texts according to their grade, was significant.

When revising Flesch’s RE formula, Kincaid et al. (1975) kept the same predictors but estimated new parameters using a criterion variable that relied on both supply words and multiple-choice test performance. For his experiment, Kincaid et al. (1975) had 531 Navy enlisted personnel take reading comprehension tests called Gates-MacGinitie Reading Test (GMRT) and passages from Rate Training Manuals. Then, multiple regression analysis was conducted using these grades as the criterion variable to validate the revised grade-level version of the Flesch (1948) formula. The Army General Classification Test scores, known for each subject, with the results of multiple regressions, constitute the conversion table from RE (Flesch 1948) to grade level (Kincaid et al. 1975), represented by the coefficients of Eq. (2).

The outcomes of our analysis reveal that the vocabulary variability (Figure 2, left panel) rose sharply in 2017 after years of relative stability and, at times, declined. This increase coincided with the change of the announcement format; i.e. shorter texts were used in combination with the provision of more detailed figures about the state of the economy.

Figure 2:

Indices of variety in vocabulary in BoI interest rate announcements. Notes: The lines and green dots represent the trend lines and index values, respectively. The TTR is calculated over each announcement. The MATTR is determined over a moving average of 100 words in each announcement.

Overall, we have 143 documents in our sample, published monthly until January 2017 and eight times a year after, and containing about 2000 words on average.

The TTR index (Figure 2, right panel) can be sensitive to text length and thus may decline with longer texts.^[6] To examine the robustness of our results, we devised a moving average of TTR values in a window that included 100 words (MATTR). Using this approach, a clear increase in text complexity since 2017 can be observed (Figure 2, left panel).

The Flesch-Kincaid text complexity index (Figure 3, left panel) exhibits relative stability with slight variance over time. Overall, a U.S. undergraduate-level education (14 years) was required to understand the BoI announcements between 2007 and 2018.

Figure 3:

Indices of text complexity in BoI interest rate announcements. Note: The lines and green dots represent trend lines and index values, respectively.

A more precise examination indicated that the average sentence length declined after 2017 (Figure 3, middle panel) from about 27 to about 22 words per sentence (although levels increased at the end of the sample), making the texts less complex. In contrast, the average number of syllables per word rose (Figure 3, right panel), increasing text complexity. In sum, the two factors offset each other such that the complexity index remained stable.

This complexity index does not consider the content of the text and any field-specific knowledge or education that may be required to understand it, regardless of education level. However, it represents an average, and aggregated effects may compensate for heterogeneities. In addition, delivering technical and professional information can also be hampered by excessively low complexity levels.

Governance changes affect interest rate communications differently over time. Under Jacob Frenkel, the length of these communications remained low (Figure 1). These communications almost doubled in length because of David Klein and the dot-com crisis. Due to the limited number of words and the high amount of redundant text, communication quality is difficult to fully assess before 2008.

With the establishment of the Monetary Policy Committee and support for the new Bank of Israel Law, the Stanley Fischer period marked the most significant changes in these communications. Over the course of his mandate, the length of texts changed from 500 to 2500 words on average (Figure 1). This increasing dynamic was also influenced by the critical role played by the GFC and financial stability in monetary policy decisions, and did not significantly change lexical diversity (Figure 2) or readability (Figure 3). However, this dynamic increased the number of syllables per word (Figure 3), indicating a slight increase in text complexity.

During the mandate of Karnit Flug, the BoI reduced the frequency of interest rate decisions from monthly to eight times a year. This change involved a drastic decrease in the interest rate communication length, switching from an average of 2500 words to an average of 1200 words per interest rate announcement (Figure 1), which also contributed to the increase in lexical diversity (Figure 2) and complexity (Figure 3). Still, overall communication quality was hardly affected as these texts were more accessible (Flesc-Kincaid declined in Figure 3) and sentence length was reduced (see the mean sentence length, Figure 3).

The changes in governors with the governance of historical (dot-com and GFC) and legal contexts (Bank of Israel Law and the establishment of the Monetary Committee) influenced the BoI’s communications.

To interpret these indices objectively, we look at the interest rate announcements made by other central banks. Figure 4 compares the complexity and linguistic variety indices of the BoI’s announcements with the indices of the communications published by the Fed and the ECB. The Fed and ECB’s announcements require an average of about 17 and 19 years of education, respectively, to be understood. The BoI announcements are relatively more comprehensible, requiring an average of about 14 years of education to be understood. Furthermore, the Fed and the ECB’s linguistic variety indices are almost three times as high as that of the BoI on average.

Figure 4:

International comparison of text complexity indices and linguistic variety. Note: Yearly averages based on interest rate announcements. Source: Benchimol, Kazinnik, and Saadon (2020).

3.2 Contextual Uncertainty

In this section, we examine the quality of the information in the interest rate announcements based on the financial context of the words they contain.^[7] Instead of analyzing divergence in tone (Sadique, Veeraraghavan, and Wachtel 2013; Tillman and Walter 2019), we build a textual uncertainty index. In the first stage, using standard dictionary-based methods that are designed to analyze financial texts, we identified words in the text that conveyed uncertainty;^[8] for example, “risk,” “uncertainty,” “volatility,” probability,” and “variable.”

In the second stage, we calculate an index of textual uncertainty, which increases as the rate of words classified in the category of uncertainty increases. The uncertainty index (Figure 5) shows that the announcements contained relatively high contextual uncertainty around the time of the GFC. The contextual uncertainty significantly declined after that, particularly after 2016, before rising in response to the global economic uncertainties that transpired toward the end of the sample period.^[9]

Figure 5:

Index of uncertainty in the BoI interest rate announcements and TA-35 VIX. Source: Bank of Israel and Benchimol, Kazinnik, and Saadon (2020).

The literature on dictionary-based classifications typically employs a unigram approach, which examines individual words independently. Our methodology aligns with this approach, acknowledging that it may lead to inaccuracies in instances where a two-term phrase, composed of independently negative words, is incorrectly classified as negative when it conveys a positive meaning (e.g. “uncertainty declined”). Despite this potential limitation, the unigram approach is well-suited for our specific corpus of relatively short texts, the structural characteristics of these texts (redundancies), and the dictionaries utilized in previous studies. Additionally, central bank announcements tend to avoid using double-negative structures and compensations between false positives and false negatives may be at play. Also, using a bigram or phrase-level classification would significantly decrease the information present in short texts, such as interest rate announcements.

We note several instances of correlation between our uncertainty index and real-world economic events. We also examine the correlation between uncertainty indices and risk levels in domestic markets, as reflected in the TA-25 VIX index.^[10] A positive Pearson correlation of 0.47 was determined across the sample, while a distance correlation of 0.54 confirmed both linear and nonlinear linkages—causality direction or the assertion that uncertainty leads to fluctuations in VIX, or vice versa, could not be inferred from the data available. The announcements may contain analyses the public is unaware of, causing market volatility. In contrast, the text may describe VIX fluctuations that would have occurred in any case, even without the announcement. Further in-depth research is required to identify causality direction.^[11]