Your turn

Create a tf-idf dataframe from the trump speeches. Do it! </div> The `sum` method that was used for the word frequencies to find the most important words is less informative here. tf-idf is meant to identify important words within a text, rather than the most important words in an entire corpus, so the meaning of summing the values across multiple texts is not apparent.However, we can use them to identify the most uniquely informative words within a given text. The most useful way to analyze the most informative words (based on tf-idf) is to switch the dataframe from wide (where each case is a speech, to long, where each row is a word-case. For our two sentence example, the result would look like this: | sentence | word| value| |-|-|-| |0| america| 0.535941 |0| is |0.535941 |0| and| 0.376623 |0 |proud |0.37662 | 0| strong| 0.376623 |0| free |0.000000 | 1| free| 0.704909 | 1| america| 0.501549 | 1| is| 0.501549 | 1| and| 0.000000 | 1| proud| 0.000000 | 1| strong |0.000000 The first sentence is about what "america" "is" while the second is about "free". In pandas, the `melt` method is used to convert a dataframe from wide to long. The first parameter is the name of the dataframe. `id_vars` is the variable or variables that will be used to identify the cases. In this case, that is `sotu_year`, which uniquely identifies each row. `value_vars` are the variables to be transposed. In this case, we only want the word variables, which are still stored in the `tfidf_vectorizer.get_feature_names()` list. {:.input_area} ```python df_long = pd.melt(df_combined, id_vars='sotu_year', value_vars=tfidf_vectorizer.get_feature_names()) ``` {:.input_area} ```python df_long.head() ```

The new dataframe is quite long. {:.input_area} ```python len(df_long) ``` {:.output .output_data_text} ``` 5669960 ``` Most of that, however, is zeros, so we can drop those cases. {:.input_area} ```python df_long = df_long[df_long['value']!= 0] len(df_long) ``` {:.output .output_data_text} ``` 356039 ``` For display purposes, it is also useful to set the word variable (which pandas renamed to `variable` as part of the `melt` as the index. {:.input_area} ```python df_long.set_index('variable', inplace= True) df_long.head() ``` Now, we can use `groupby` and `nlargest` to find the most important word, as measured by tf-idf value, for each year. {:.input_area} ```python df_long.groupby('sotu_year')['value'].nlargest(5) ``` {:.output .output_data_text} ``` sotu_year variable 1790 kentucky 0.117869 militia 0.116685 requisite 0.111104 consuls 0.105274 attachment 0.099320 1791 states 0.194833 united 0.184807 subscriptions 0.137851 objects 0.124032 provision 0.112637 1792 chickamaugas 0.139889 provision 0.126599 proper 0.100720 information 0.099347 subject 0.097114 1793 states 0.302194 united 0.278213 shall 0.108190 warmest 0.106906 liberate 0.104253 1794 militia 0.210568 pennsylvania 0.208337 states 0.167961 inspector 0.143016 united 0.133176 1795 indians 0.196307 tranquillity 0.151404 gentlemen 0.135764 violences 0.120053 situation 0.096290 ... 2013 jobs 0.275742 tonight 0.190875 america 0.136807 let 0.127932 families 0.122968 2014 help 0.197449 jobs 0.191839 americans 0.174650 america 0.161544 cory 0.151007 2015 america 0.181501 childcare 0.173489 rebekah 0.173489 new 0.160102 jobs 0.150926 2016 isil 0.233942 america 0.160145 voices 0.148003 world 0.137319 americans 0.125783 2017 megan 0.217931 america 0.173996 american 0.167585 tonight 0.154645 ryan 0.154251 2018 tonight 0.249086 americans 0.178473 otto 0.148802 america 0.147480 american 0.143322 Name: value, Length: 1140, dtype: float64 ``` I'm not that familiar with the early years of the republic, so the analysis can be limited to the most recent inaugaural years. {:.input_area} ```python inaugaration_years = [1977, 1981, 1989, 1993, 2001, 2009, 2017] df_inaugaration = df_long[ df_long['sotu_year'].isin(inaugaration_years) ] ``` {:.input_area} ```python df_inaugaration.groupby('sotu_year')['value'].nlargest(5) ``` {:.output .output_data_text} ``` sotu_year variable 1977 energy 0.189392 president 0.163595 strategic 0.154868 today 0.121060 congress 0.118501 1981 1980 0.226320 administration 0.164795 program 0.152728 programs 0.140324 soviet 0.132398 1989 tonight 0.284489 america 0.218648 budget 0.174209 let 0.147870 drugs 0.144559 1993 jobs 0.209817 people 0.184011 deficit 0.163053 investment 0.139919 percent 0.138959 2001 terrorists 0.265787 tonight 0.222926 terror 0.177895 americans 0.153074 qaida 0.147483 2009 know 0.150395 jobs 0.146521 lending 0.143254 recovery 0.135559 health 0.134195 2017 megan 0.217931 america 0.173996 american 0.167585 tonight 0.154645 ryan 0.154251 Name: value, dtype: float64 ``` {:.input_area} ```python df_inaugaration.groupby('sotu_year')['value'].nlargest(8).plot.barh(figsize=(5,20)) ``` {:.output .output_data_text} ``` <matplotlib.axes._subplots.AxesSubplot at 0x121b09668> ``` {:.output .output_png}  ### Bonus Visualization - TSNE The TSNE algorithm in scikit-learn follows the same logic of as the vectorizers: 1. Establish the parameters. In this case, we want to reduce or data to two dimensions. 2. Fit the model on the data. 3. Transform the data to the two dimensions. As noted above, one major use for term frequency and tf-idf arrays is as an intermediary step for subsequent analysis, such as text classification or topic models. These arrays are often very wide, meaning they have hundreds or thousands of columns, each associated with a distinct word. Analysis often use a second intermediary step in order to reduce the number of variables, using techniques such as principle component analysis (PCA). TSNE, t-distributed stochastic neighbor embedding, is an alternative algorithm for data reduction that is designed for visualization. It is usually used to reduce the number of variables to two, which become the x and y values for graphing. While the co-ordinates themselves have no meaning, similar cases are presented as clustered together. The TSNE algorithm in scikit-learn follows the same logic of as the vectorizers: 1. Establish the parameters. In this case, we want to reduce or data to two dimensions. 2. Fit the model on the data. 3. Transform the data to the two dimensions. For many scikit-learn functions, steps 2 (`fit`) and 3 (`transform`) can be combined with the `fit_transform` method, which we will use here. {:.input_area} ```python from sklearn.manifold import TSNE ``` {:.input_area} ```python tsne = TSNE(n_components=2) ``` {:.input_area} ```python coordinates = tsne.fit_transform(tfidf_df) ``` {:.input_area} ```python coordinates[:10] ``` {:.output .output_data_text} ``` array([[-15.743554 , 15.046603 ], [-13.368821 , 14.946587 ], [-12.693366 , 15.6549425], [ -9.696101 , 12.482419 ], [ -8.252132 , 12.2527685], [-14.5627365, 15.672341 ], [-14.645704 , 17.25603 ], [-13.702745 , 19.313482 ], [-14.084031 , 19.285786 ], [-12.786504 , 19.198757 ]], dtype=float32) ``` These coordinates can be stored in a dataframe. {:.input_area} ```python tsne_df = pd.DataFrame(coordinates, columns=['tsne_y', 'tsne_x']) ``` {:.input_area} ```python tsne_df.info() ``` {:.output .output_stream} ``` <class 'pandas.core.frame.DataFrame'> RangeIndex: 230 entries, 0 to 229 Data columns (total 2 columns): tsne_y 230 non-null float32 tsne_x 230 non-null float32 dtypes: float32(2) memory usage: 1.9 KB ``` Once in a dataframe, the tsne co-ordinates can be plotted. I use the `fig_size` option to ensure the result are square. {:.input_area} ```python tsne_df.plot.scatter(y='tsne_y', x = 'tsne_x', title ='t-SNE Plot of Presidential Addresses', figsize=(5,5) ) ``` {:.output .output_data_text} ``` <matplotlib.axes._subplots.AxesSubplot at 0x117826a90> ``` {:.output .output_png}  Without year variables, this doesn't tell us much about the patterning of speeches. We can copy over the year variable from the original dataset. {:.input_area} ```python tsne_df['year'] = sotu_df['sotu_year'] tsne_df['year'].describe() ``` {:.output .output_data_text} ``` count 230.000000 mean 1904.334783 std 66.263772 min 1790.000000 25% 1847.250000 50% 1904.500000 75% 1961.000000 max 2018.000000 Name: year, dtype: float64 ``` Ideally, the graph would allow you to select an individual point to find out the year, the president, and the text of the speech. This is possible with some graphing libraries, such as [bokeh](https://bokeh.pydata.org/en/latest/) or [plotly](https://plot.ly), but beyond the scope of this chapter. Rather than labeling each point, we can bin them using the pandas `cut` method. For this case, I create ten different groups stored in the new variable `era`. {:.input_area} ```python tsne_df['era'] = pd.cut(tsne_df['year'], 10) ``` [seaborn](https://seaborn.pydata.org) is a popular visualization library that makes the standard python plots prettier and easier to use. It also has the most convient method for coloring points in a scatterplot using a third variable, such as era. By convention, seaborn is imported as `sns`. {:.input_area} ```python import seaborn as sns ``` `pairplot` can be used to produce a matrix of correlations, but we are only interested in our t-SNE coordinates. Note the syntax is slightly different here, as the x and y are just the column names and the dataframe name is set through the `data` parameter. {:.input_area} ```python sns.pairplot(x_vars = 'tsne_x', y_vars = 'tsne_y', data = tsne_df, hue = 'era', size=6) ``` {:.output .output_data_text} ``` <seaborn.axisgrid.PairGrid at 0x13e1e1ac8> ``` {:.output .output_png}  This plot suggest that the language of presidential address is largely driven by temporal trends, rather than by party or region. A break seems to have occurred around the early 1920. A close inspection of this time period, through an examination of word frequencies or other quantitative or qualitative methods of text analysis, might suggest what the substance of the shift identified here. ### Bonus: Bigrams You might be interested not only in the frequency of certain words, but also word pairs with a distinct meaning, such as African American or United States. These word pairs are called bigrams, are more generally, ngrams. If you have are familiar with your corpus and have a finite number of ngrams you want to handle, the best strategy is to preprocess the text yourself by writing a function that turns each of your ngrams into a single word. {:.input_area} ```python def my_ngrams(text): '''Replace ngrams in a string with unigrams''' text = text.replace("African American", "AfricanAmerican") text = text.replace("United States", "UnitedStates") text = text.replace("House of Representatives", "HouseofRepresentatives") return text sotu_df['clean_sotu_text'] = sotu_df['sotu_text'].apply(my_ngrams) ``` The above function does a series of replaces, one for each ngram that you want to fix. Alternatively, you could create a list of the ngrams and then loop through them. This method scales up to larger lists much better and avoids lines of duplicate code. It also takes advantage of the fact that the replacement text is the original text with the spaces removed (or replacing `' '` with `''`). {:.input_area} ```python ngrams = ['African American', 'United States', 'House of Representatives'] def my_ngrams2(text): '''Replace ngrams in a string with unigrams''' for ngram in ngrams: replacement = ngram.replace(' ','') text = text.replace(ngram, replacement) return text sotu_df['clean_sotu_text'] = sotu_df['sotu_text'].apply(my_ngrams2) ``` Subsequent text analysis would be done on the new variable `clean_sotu_text` rather than the original, untransformed version. Alternatively, you may want to identify ngrams in a more algorithmic way, either to identify candidates for your custom function or to use in your analysis. Both of scikit learns text vectorizers include an option to identify ngrams, `ngram_range`. It takes a pair of values, the first is the minimum value (where 1 is a single word) and the second is the maximum number. So `(1,2)` would construct a vocabulary of all unigrams and bigrams, while `(3,3)` would be only trigrams. To identify the most common bigrams, we can rerun the analysis above, but this time setting with `ngram_range = (2,2)` as a parameter for the `CountVectorizer`. {:.input_area} ```python vectorizer = CountVectorizer(lowercase = True, ngram_range = (2,2), stop_words = 'english' ) ``` {:.input_area} ```python bi_tf = vectorizer.fit_transform(sotu_df['sotu_text']) ``` {:.input_area} ```python bi_tf_df = pd.DataFrame(bi_tf.todense(), columns = vectorizer.get_feature_names()) ``` {:.input_area} ```python bi_tf_df.sum().sort_values(ascending=False)[:10] ``` {:.output .output_data_text} ``` united states 4494 000 000 1004 fiscal year 748 great britain 517 federal government 402 american people 389 fellow citizens 293 june 30 287 report secretary 284 house representatives 274 dtype: int64 ``` Note that 'house representatives' likely was 'House of Representatives" in the original text, but the stop words removal happened prior to the ngram process. Replicating the process with both unigrams and ngrams: {:.input_area} ```python vectorizer = CountVectorizer(lowercase=True, ngram_range = (1,2), stop_words = 'english' ) bi_tf = vectorizer.fit_transform(sotu_df['sotu_text']) bi_tf_df = pd.DataFrame(bi_tf.todense(), columns = vectorizer.get_feature_names()) bi_tf_df.sum().sort_values(ascending=False)[:10] ``` {:.output .output_data_text} ``` government 7081 states 6476 congress 5003 united 4820 united states 4494 people 4076 year 3997 country 3436 great 3258 public 3084 dtype: int64 ``` Subsequent sections will discuss ways to decide how to algorithmically decide what are the best values for different vectorizer options, such as ngrams range or whether to include stop words or not. ### Bonus: Stemming Should you count 'Americans' the same as "American"? Should "runs" and "run" be combined in your count? Having both singular and plural versions of the same noun or multiple variants of a verb can often clutter your analysis, particularly when looking at the most frequent words. In a large, diverse corpus, an article about "immigration" probably should be grouped with one about "immigrants", although this relationship might be obscured because of word variations. Two common strategies for handling this issues are stemming and lemmatization. Stemming is a rule-based method to chop off the end of a word, such as removing "ing" at the end of a word so that "falling" becomes "fall" or the "d" for words that fit a pattern of constant followed by a "ed", such as "conflated" becomes "conflate" but "feed" does not become "fee". In contrast, lemmatization is a process of identifying the dictionary form of a word, or lemma. So while a stemmer might turn "communities" into "commun", the lemma is "community". Lemmatization works best with knowledge of the part of speech, as the lemma of the verb "saw" is "see", while the lemma of the noun "saw" is "saw". Both methods have strengths and weaknesses. Stemming is fast and bins many related words together, but it can be too aggressive in chopping and the stems are often non-obviously interpretable. In contrast, lemmatization can be quite computationally intensive, especially when the text need to be part-of-speech tagged first. In practice, neither is commonly used, although they are useful tools to have knowledge of. Both stemming and lemmatazation are available in Python through NLTK, the Natural Language Tool Kit. There are several variants of each available, but the most common are the `PorterStemmer` and the `WordNetLemmatizer`. Note: If this is the first time you are using either, you will need to download auxilary files for them first. ``` import nltk nltk.download('punkt') nltk.download('wordnet') ``` {:.input_area} ```python from nltk.stem.snowball import PorterStemmer from nltk.stem.wordnet import WordNetLemmatizer ``` {:.input_area} ```python stemmer = PorterStemmer() ``` {:.input_area} ```python stemmer.stem('falls') ``` {:.output .output_data_text} ``` 'fall' ``` {:.input_area} ```python stemmer.stem('organizing') ``` {:.output .output_data_text} ``` 'organ' ``` {:.input_area} ```python lemmer = WordNetLemmatizer() ``` {:.input_area} ```python lemmer.lemmatize('falls') ``` {:.output .output_data_text} ``` 'fall' ``` {:.input_area} ```python lemmer.lemmatize('organizing') ``` {:.output .output_data_text} ``` 'organizing' ``` As you can see, both have similar syntax, although the lemmatizer also has the capacity to accept a part-of-speech tag. If defaults to assuming things are a noun. If the word is not found it its dictionary as a noun, it returns the original word. However, we can produce the verb lemma. {:.input_area} ```python lemmer.lemmatize('organizing', 'v') ``` {:.output .output_data_text} ``` 'organize' ``` Incorporating stemming or lemmatization in vectorizer requires passing a custom tokenizer. The standard tokenizer takes a text string, strips the punctuation and other one-character words, and returns a list of words. So to create a custom stemming tokenizer, we need a function that does the same but also stems each word. In my `tokenize`, I use `word_tokenize` from nltk to split the sentences into words. The second line stems each word, excluding those tokens that are punctuation or stop words. This second second line uses list comprehension, a concise way to speed up list creation without using `for` loops. The one I employ below is equivalent to: ``` stems = [] for w in words: if keep(w) == True: s = stemmer.stem(w) stems.append(s) ``` \[Personally, I find it easier to read a `for` loop, but list comprehensions are fast and pithy.\] {:.input_area} ```python from nltk import word_tokenize from nltk.corpus import stopwords from string import punctuation stemmer = PorterStemmer() stop_words = stopwords.words('english') def keep(word): '''Test whether a token is a stop word or punctuation''' if word in punctuation: return False elif word in stop_words: return False return True def tokenize_and_stem(text): '''Tokenize and stem a text returning a word list.''' words = word_tokenize(text) stems = [stemmer.stem(w) for w in words if keep(w) == True] stems return stems ``` We can test that `tokenize` returns what we expect it to: {:.input_area} ```python tokenize_and_stem('America is strong because Americans work hardest under adversity.') ``` {:.output .output_data_text} ``` ['america', 'strong', 'american', 'work', 'hardest', 'advers'] ``` The `tokenize` function can be added as an option for a count vectorizer. {:.input_area} ```python stem_vectorizer = CountVectorizer(tokenizer = tokenize_and_stem) ``` {:.input_area} ```python stem_tf = stem_vectorizer.fit_transform(sotu_df['sotu_text']) stem_tf = pd.DataFrame(stem_tf.todense(), columns = stem_vectorizer.get_feature_names()) ``` {:.input_area} ```python stem_tf.sum().sort_values(ascending=False)[:20] ``` {:.output .output_data_text} ``` state 8805 govern 7955 year 6198 nation 6079 congress 5019 unit 5011 countri 4445 peopl 4262 upon 3957 would 3813 -- 3651 american 3513 law 3467 may 3400 time 3323 great 3309 must 3272 public 3146 made 3143 power 3093 dtype: int64 ``` Big winners here is "state" as the combination of "states" and "state", along with a few "stated". "Nation" combines "nation" and "nations" plus "national" and a few "nationality", but no "nationwide", which stems to "nationwid".