## Authenticating Twitter
The chunk of code below is used to authenticate your Twitter developer account. You'll need to navigate here: https://developer.twitter.com/en.html and follow the instructions to get a developer account, setup an app for access tokens, and generate tokens and secret keys to sign in. This is totally optional! I've included the code here (minus my secret access keys!) to show you how it might be done.

```{r}
library(twitteR)

consumer_key <- ""
consumer_secret <- ""
access_token <- ""
access_secret <- ""

setup_twitter_oauth(consumer_key, consumer_secret, access_token, access_secret)

ls_twitter <- searchTwitter("#Impeachment -filter:retweets", n = 1000, lang = "en")
df_twitter <- twListToDF(ls_twitter)
```

## Working with Twitter data
Our Twitter data is contained in a dataframe. Let's take a look at what's included:

```{r}
library(tidyverse)
library(tidytext)
library(textdata)

df_twitter <- readRDS("twitter-data.rds")

names(df_twitter)
head(df_twitter$text)
```

Let's begin our text analysis with some very basic operations from TidyText. The `unnest_tokens` function is a *tokenizer* function. It splits text up into simple objects based on the first input --- this can be a word, sentence, ngram, or other options. It generates a dataframe containing the word (and we've included the id of the document the word is from here, as well).

```{r}
tweet_words <- df_twitter %>% select(id, text) %>% unnest_tokens(word,text)
head(tweet_words)
```

Now, we'll generate a count of the words, sort by the number of times the word occurs, and then plot the top 15 words in a bar plot.

```{r}
tweet_words %>% count(word, sort = T) %>% slice(1:15) %>% 
  ggplot(aes(x = reorder(word, n, function(n) -n), y = n)) + 
  geom_bar(stat = "identity") + 
  theme_light() +
  theme(axis.text.x = element_text(angle = 60, hjust = 1)) + 
  xlab("Words")
```

Obviously, something is off here! Text data can be *really* noisy. There are lots of artifacts floating around out there. Some to look out for include:
 - URLs
 - Non-unicode characters --- always make sure you keep your text encoding straight, be it ASCII, UTF-8, or something else.
 - Stop words
 
What is a *stop word*? These are words that you don't want to include in your analysis, and are often the most frequently used words in the language your text data is in. We should get rid of them before ranking thes words properly. Luckily, TidyText comes with built in stop words! You can invoke them with `stop_words`. Take a look below:

```{r}
head(stop_words)
```

Now, we have stop words that are domain-specific. That is, stop words that are unique to our corpus of text. For the Twitter data, these include `https`, `t.co`, and likely others --- but for now we'll just cull the ones we can see above. (Question: why might we include the word "impeachment" here?)

```{r}
personal_stop_words <- stop_words %>% select(-lexicon) %>% 
  bind_rows(data.frame(word = c("https", "t.co","impeachment")))

```

Now, we'll get rid of words from the dataframe that match these stop words.

```{r}
better_words <- tweet_words %>% anti_join(personal_stop_words)
```

```{r}
better_words %>% count(word, sort = T) %>% slice(1:15) %>% 
  ggplot(aes(x = reorder(word, n, function(n) -n), y = n)) + 
  geom_bar(stat = "identity") + 
  theme_light() +
  theme(axis.text.x = element_text(angle = 60, hjust = 1)) + 
  xlab("Words")

```

## Sentiment analysis and TF-IDF

Now, just the word count doesn't tell us all that much. Let's turn to a "better" count measure --- the term frequency inverse document frequency.

```{r}
idf_words <- df_twitter %>% select(id, text) %>% 
  unnest_tokens(word,text) %>% count(id, word, sort = T)

better_idf_words <- idf_words %>% anti_join(personal_stop_words)

tweet_length <- better_idf_words %>% group_by(id) %>% summarize(total = sum(n()))

better_idf_words <- left_join(better_idf_words, tweet_length)
```

OK, let's get our tf-idf. Use the `bind_tf_idf` function to have R do this for you automatically. The first input is the word itself, the second is the id of the document, and finally the count of words per document.

```{r}
word_tfidf <- better_idf_words %>% bind_tf_idf(word, id, n)

word_tfidf %>% arrange(desc(tf_idf))
```

There are some interesting results here. Why might our weights have produced these results?

This is a look into the fascinating world of linguistics --- one reason we're getting these results is because of *Zipf's law of abbreviation*: the more frequently a word is used, the "shorter" the word tends to be. And since Twitter has a character limit, there is a ton of downward pressure on word length. 

Based on this law and the results we see, tf-idf is actually not the best thing to use here. We're OK with just our original word count.

### Sentiment Analysis

```{r}
sentiments <- get_sentiments("nrc")

df_sentiments <- better_words %>% left_join(sentiments)

df_sentiments_filtered <- df_sentiments %>% filter(!is.na(sentiment)) %>% group_by(sentiment) %>% summarize(n = n())

df_sentiments_filtered %>% 
  ggplot(aes(x = reorder(sentiment, n, function(n) -n), y = n)) + 
  geom_bar(stat = "identity") + 
  theme_light() +
  theme(axis.text.x = element_text(angle = 60, hjust = 1)) + 
  xlab("Sentiments")

```
