Words Count, But Context Counts More
Of course words count....especially in the policy context. In a Distributed Age turbo-charged by social media, it is no longer true that "words can never hurt me." When words are used by public policy officials, words take on additional, action-oriented attributes which are not present when private sector entities utter the same words.
In other words (pardon the pun), policy language is profoundly more predictive of future decisions than most people appreciate because it articulates action. Superforecasters know this, which is why they focus on minute shifts in position. The question is whether the superforecasting process can be automated and scaled. We believe, based on experience, that the answer is yes. We recently had some fun with this idea in this blogpost and YouTube video.
The trick is to use the right tools to convert the unstructured policy language into structured data. Natural Language Processing (NLP) and Natural Language Understanding (NLU) are the obvious candidates for the task. Regrettably, the very context that drives meaning in political language creates massive computational challenges for even the most advanced NLP/NLU utilities. This post explains why.
Free Speech 101 -- Basic Concepts
Let's start with a counterintuitive and deliberately provocative statement: there is no such thing as 100% free speech in a public setting. In fact, there is not such thing as unrestricted freedom in a public setting.
Enlightenment thinkers sparked revolution by maintaining that the natural state for humans (OK, at the time, white men who owned land) was absolute -- but NOT unlimited -- freedom. Maximum freedom was to be found on one's own land. To this day, in the United States, this concept underpins self-defense laws in one's home. A good high-level historical overview of the "castle doctrine" can be found in this Wikipedia entry.
We will let others continue the centuries old polemical debate on whether the consequent enshrinement of private property created a "tragedy of the commons." This post does not pass value judgements; it merely explains how we got to where we are. The history matters in order to understand well the technology challenges that arise when we ask bots and algorithms to parse policy language.
John Locke and others made clear that people need to coexist. Your freedom is not entitled to disrupt someone else's ability to enjoy freedom. When your actions disturb others' rights, agreement must be reached on how competing rights should be shared along the overlap. This concept continues to underpin modern laws on pollution emanating from private property as well as traffic safety and other laws governing shared spaces.
Similar standards apply to speech.
The United States Constitution, in the First Amendment, famously establishes a right of free speech without limitation. Among other things, the idea was that if someone has the right to say what they think, this serves as a release valve that makes violent action less likely. Of course, Eighteenth century society did not have Twitter or Facebook or Reddit.
Throughout the twentieth century (long before the rise of social media), the Supreme Court interpreted the First Amendment to include boundaries on free speech that are conceptually similar to the limits on freedom that operate on private property. The key is the potential for physical harm. The Supreme Court has made clear that the limit of free speech by private citizens ceases to operate if the speech seeks to incite illegal/dangerous action and will likely generate actual illegal/dangerous action. Hate speech legislation extends this concept by defining certain categories of speech that are automatically considered to be so powerful that they are likely to incite odious actions.
We will leave aside for another day the question of whether banning speech increases or decreases the likelihood of odious or dangerous activity. We bypass altogether the amazing and difficult debate underway about whether and how certain speech should be suppressed on social media platforms by democratically elected governments or private parties.
The key point for this blogpost is that all these standards were created with the private citizen in mind.
Why Policy Language is Different
Policymakers enjoy far more freedom of speech than private citizens.
Article I, Section 6 of the Constitution insulates from legal action "any Speech or Debate" by Members of Congress. We will let constitutional scholars debate whether the punctuation in the text applies to Congressional speech the same exceptions that apply to arrest ("except treason, felony and breach of the peace") while in transit to or from the Congress. The insulation of Congressional speech, among other things, is the foundation for immunity from prosecution for libel or slander any statements made by Members of Congress on the floor of the House or Senate.
Why do Members of Congress enjoy these immunities? Because an elected official is deemed to be speaking not just for himself or herself. The official is deemed to be speaking on behalf of constituents. The official serves as a megaphone; the size of the megaphone can be considered to be the size of the constituency that gets to determine at specific intervals whether the individual continues to have the honor of articulating opinions on behalf of the constituency.
This makes policy language different from private speech in rather profound ways.
Among other things, a policy official is constantly aware that his or her words will generate ripple effects and have concrete consequences for a large number of people if not whole industries and economies. Policymakers weigh their words deliberately, even when it seems they are acting impromptu..
What policymakers say matters. Alot. Every word implies action and holds the potential for generating a reaction.
Elected policymakers predominantly speak in simple language laced with normative values that resonate with voters. Even when discussing technical arcane issues of monetary policy, regulatory capital, cybersecurity, blockchains, etc., they prioritize language that can be understood by voters who hold the power to turn them out of office in the next election if they say and do the wrong thing.
The same is true regarding speech by appointed officials (e.g., regulators, central banks), but with a twist. These officials cater to a technical constituency more than voters. Insulated from ballot box pressures, these officials effectively speak their own dialect to a well-informed constituency that speaks the same language. For example, bond market traders are fluent in "Fedspeak." Banking regulators use "Basel-speak."
Every word uttered by appointed policymakers also matters and also implies action. But the language must be decoded to be understood by non-experts. For an in-depth description of how to distinguish rhetoric from action within the official sector, see this blogpost from Interactive Brokers.
How Policy Language Creates Problems for Natural Language Processing
The good news is that technology in principle is well-suited to deliver translation mechanisms. . The age of data is upon us, with multiple methods for converting unstructured verbal data (words, speeches, text) into structured integers that can be visualized. Sophisticated algorithms can identify hidden correlations within large amounts of structured data that would take weeks or longer for humans to identify.
The bad news is that every mechanism we have seen so far for parsing language focuses on retail, private language from which large amounts of data can be harvested from online sources. Even worse, the vast majority of NLP focuses on predicting the next word in a sentence in order to deliver superior, automated customer experiences via web-based chatbots or concierge-like robots in hotels, houses, and cars. Automated text generation software in the journalism field generates stories based on statistics (sports, corporate earnings) where the principal author is a machine whose only interaction with meaning is defined in terms of volume. Higher stats (e.g., earnings) are good, so a pre-set lexicon of favorable phrases are triggered for delivery.
One cannot apply retail-focused NLP/NLU utilities to the policy language context and expect good results in terms of efficient analysis much less predictive analytics.
Most efforts to create structured data from language ultimately involve counting words. Classic examples include word clouds and sentiment analysis. It is rather rudimentary. How many times is a specific word used? If certain positive or negative words are used, then the speaker or writer must have a positive or negative normative view of the issue at hand and frequency determines the strength of the sentiment.
The approach will deliver many false positives in the policy context. Consider these examples:
Central bank speeches will always register a high incidence of the words “monetary policy” and negative sentiment regarding “inflation.”
A financial regulation speech can be expected to have a high incidence of the words “financial stability.” Spoiler alert: financial regulators always say good things about financial stability, but that does not mean they believe they have achieved it. The NEVER say good things about contagion risk or systemic risk. They rarely actually use the term “financial instability,” preferring instead “financial volatility."
A data privacy supervisor can be expected to have a high incidence of the words “data privacy” and “data protection.” Spoiler alert: these terms are not synonyms.
Counting words tells the strategic policy analyst nothing interesting.
Sentiment analysis does not help. The task of identifying sentiment from public policy statements is a minefield.
It is true that all people can agree at a high level that certain public policy goals are desirable (e.g., freedom, fairness). But when we start talking about individual policies, views can differ dramatically on whether the policy as a good one...or not. This informs how one perceives the attitude of a speaker.
Programming algorithms to identify sentiment in the public policy context creates a high risk that the programmer’s preferred perspective (or bias) is hardwired into the analysis from the beginning.
Consider Brexit. Applying sentiment analysis to statements made regarding Brexit only provides a megaphone to the loudest or most prolific contributors to the debate. Identifying sentiment accurately amid high levels of sarcasm and specific idiomatic uses of language in England presents additional challenges.
The situation does not improve if the programming priority seeks simply to identify the sentiment of the speaker, rather than normative value. Is a policymaker for or against a specific policy? Even this can be difficult to discern using sentiment analysis.
Consider the Member of Congress exercising his or her protected Article I free speech right on the floor of the House or Senate. The Member can spend twenty minutes expressing negative sentiment about a particular law that is up for a vote...but the last sentence will be "and yet I will vote in favor of..." Word counting and sentiment analysis will have all pointed towards a nay when instead the Member was positioning for an aye.
Context and Concepts Matter More Than Words -- the NLP Frontier
Technology evangelists will pounce at this stage to declare that this is why machine learning and artificial intelligence provide a solution. Taking a kitchen sink approach, these people will feed an ML/AI system large amounts of public policy data (e.g., the Congressional Record or the Federal Register or the Official Record in Europe) and then sit back and hope for the best.
Let's glide past the cheap shots about "garbage in/garbage out." The more important critique is this: an automated system with the average intelligence of an elementary school kid will certainly count and organize words faster than an average human or even an average lawyer. But unless the system has first been trained to identify equivalent concepts buried within the technical language, the system will not generate meaningful or reliable much less actionable analysis.
There are no shortcut here, people.
As Cassie Kozyrkov (Google’s head of decision science) recently observed in this Medium post: “we…exaggerate the difference between our innate information processing and the machine-assisted variety. The difference is durability, speed, and scale… but the same rules of common sense apply in both. Why do those rules go out the window at the first sign of an equation?”.
NLU systems focus on understanding context within paragraphs and other lexical components. They hold more promise.. But even these systems are relatively new. As far as we know, these systems have not been fully tested on public policy language separate from retail, vernacular language.
We are always on the lookout for solid new research in this area. Interested readers are cordially invited to contribute to the conversation with credible, relevant research.
So yes, policy words count...but context and concepts matter more. Tracking concepts through the noise of the news cycle (and even beyond the news cycle) generates a far more meaningful mechanism for parsing the natural language of policymakers.
This is why we patented our MetaData tagging process back in 2011. This is also why we are holding back on deploying ML/AI techniques. Significant research is needed on whether one-size-fits-all when it comes to NLP techniques. As this post indicates, the answer to this question is likely NO. Training data matters quite alot, of course. We expect that the data we have assembled this year (2019) will provide a good foundation for targeted research in 2020.
Along the frontier, we expect (and hope) that a data-driven approach to policy language will deliver a fair amount of disruption in the policy intelligence business. Increased transparency about policy choices and trajectories means decreased exposure to headline risk for traders. Perhaps more importantly, transparent, objective data empowers individuals to make better, more rational choices about decisions from public officials based on cold, hard facts rather than emotion (not to mention innuendo, rumor, and worse). Join us as we explore this frontier.