#The emotional arc of presidential speeches

##Introduction

After Joe Biden's State of the Union address on March 1, I wondered whether these speeches follow a pattern. Do they have the same emotional shape, with the threats first and the hopes at the end? And does it change from one president to the next?

To find out, I scored every sentence of the 21 annual addresses to Congress since 2002, from George W. Bush to Joe Biden, and drew each speech as a strip of colour, from its first word to its last.

##Data

The Miller Center of the University of Virginia publishes the transcripts of presidential speeches, one page per speech. I took every State of the Union address from 2002 to 2022, plus the three addresses to a joint session of Congress that new presidents give in their first year instead (2009, 2017, 2021): 21 speeches, between 3,800 and 7,900 words each.

import re
import requests
from bs4 import BeautifulSoup

def get_transcript(url):
    soup = BeautifulSoup(requests.get(url).text, 'html.parser')
    text = soup.find('div', class_='transcript-inner').get_text('\n')
    text = re.sub(r'\[.+?\]', '', text)                           # stage directions
    text = re.sub(r'\((?:[Aa]pplause|[Ll]aughter)\.?\)', '', text)
    return text.strip()

##Scoring sentences

For the sentiment, I used VADER, a model built for social media posts but which works well on spoken English. It relies on a dictionary of words rated by people from very negative to very positive, plus rules for negations ("not good"), intensifiers ("very good") and punctuation. It gives each sentence a compound score between −1 and +1.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()

sentences = re.split(r'(?<=[.!?])\s+(?=[A-Z"\'])', re.sub(r'\s+', ' ', text))
scores = [analyzer.polarity_scores(s)['compound'] for s in sentences]

A sentence-by-sentence score jumps around too much to read. So each word takes the score of its sentence, and a moving average over 100 words smooths the result: long sentences weigh more than short ones, and the curve follows the passages of the speech rather than individual sentences.

import numpy as np
import pandas as pd

def word_series(df, window=100):
    words = pd.Series(np.repeat(df['score'].to_numpy(), df['words'].to_numpy()))
    return words.rolling(window, center=True, min_periods=1).mean()

Here is the result for the 2002 address, the first after September 11: one bar per sentence, as wide as its number of words, and the moving average in black.

The speech opens on the war in Afghanistan, the "axis of evil" and the threat of terrorist attacks: its first 1,400 words are mostly negative. Then, about a third in, it turns to the economy, education and volunteering, and stays positive until the end.

Here is that turn, around the 1,400th word, with each sentence highlighted in the colour of its score: red when negative, blue when positive, white when neutral.

##All the speeches

The same moving average for the 21 speeches, each as a strip that goes from the first word (left) to the last (right), whatever its length:

A few patterns stand out:

• The ending is always positive. All 21 speeches finish in blue: the closing minutes are about the strength of the nation, its future, and God's blessing.
• Wars set the red zones under Bush. In 2002, the beginning; in 2003, the second half, devoted to Iraq and its weapons ("the gravest danger facing America and the world"); in 2007, the middle, on the surge of troops in Iraq.
• Obama's speeches are the most evenly positive, with short red patches, mostly on terrorism (al-Qaeda, then ISIL in 2016).
• Under Trump, the red passages move to crime and immigration, with the stories of victims of "criminal aliens" in 2019 and 2020, and North Korea in 2018.
• Biden's red patch in 2021 is gun violence, around 80% of the speech.

##By president

Averaged over a whole speech, the differences are smaller than the strips suggest:

Obama has the highest average (0.21), ahead of Trump (0.18), Bush (0.16) and Biden (0.13). Each president's lowest point is easy to explain: 2003 for Bush, on the eve of the war in Iraq; 2010 for Obama, after the financial crisis; 2019 for Trump, just after the government shutdown over the border wall. Biden's two speeches are steady, but on the low side: more of their sentences are neutral than in any other president's speeches.