Text mining

Because the data is loaded already, let’s see how we would process that. Working with natural language or natural language processing usually requires extensive data cleaning and preparation of the raw data. The preparation steps include

• defining words splits (tokenization) and terms (n-grams)
• standardization of the words, e.g. through typo dictionaries and stemming/lemmatization
• filtering of stop words and punctuation, maybe also based on a pre-defined dictionary

These steps and the word lists/dictionaries depend on the language and the goal of the analysis. For example, it is probably a bad decision to remove punctuation including smileys from tweets.

We will be using package tm to do these preprocessing tasks, the standard text mining package in R (at the moment) although I’ll check out tidytext when I find the time. The package has several convenient functions and and structure to work efficiently on large corporae: An object of text elements can be defined as a Corpus object and function tm_map() (see base function Map()) can consequently be used to map any function to each document within the corpus.

# We will use package tm (text mining)
library(tm)

## Loading required package: NLP

# Package tm provides infrastructure to read in text from 
# different source, e.g. a folder of files, an XML file, 
# or a vector, as in this case
review_texts <- reviews$text[1:100]
reviews_source <- VectorSource(review_texts)
# After we have specified our vector as a source,
# we can read the reviews into a corpus, which contains
# the text and meta data for each document
corpus <- SimpleCorpus(reviews_source)

# Ultimately, we aim to break the texts up into small pieces called tokens,
# words or terms, and count how often these appear in each document.
# Since very similar words come in different forms, e.g. pizza/Pizza/pizzas/pizza!,
# we standardize the text elements before counting.

# Transform all letters to lower-case
# Function tm_map applies the function argument to all documents in the corpus
corpus <- tm_map(corpus, content_transformer(tolower))
# Remove all punctuation characters
replaceCharacter <- content_transformer(function(x, pattern, replacement)
    gsub(pattern = pattern,replacement = replacement, x))
corpus <- tm_map(corpus, replaceCharacter, "'", "")
corpus <- tm_map(corpus, replaceCharacter, "[[:punct:]]", " ")
# Here we make a decision to separate pizza-place into pizza place
# but wouldn't into wouldnt

# Reduce all whitespace to one and delete line breaks, etc.
corpus <- tm_map(corpus, stripWhitespace)
# Remove words without semantic content, like 'and' or 'it'
# You see that stopwords are language specific 
head(stopwords("english"), 10)

##  [1] "i"         "me"        "my"        "myself"    "we"       
##  [6] "our"       "ours"      "ourselves" "you"       "your"

corpus <- tm_map(corpus, removeWords, stopwords("english"))
# Reduce all words to their word stem 
# Lemmatization would require non-R dictionaries, e.g. TreeTagger 
corpus <- tm_map(corpus, stemDocument, "english")
# Here we choose to ignore that there are French and German reviews in the corpus

# We can now caluclate the document-term matrix
# During the process, we ignore terms that occur in less than five reviews
# because we expect them to be relevant for only a minor number of observations
dtm <- DocumentTermMatrix(corpus, control = list(bounds = list(global = c(5,Inf))))
inspect(dtm[1:10, 1:20])

## <<DocumentTermMatrix (documents: 10, terms: 20)>>
## Non-/sparse entries: 40/160
## Sparsity           : 80%
## Maximal term length: 7
## Weighting          : term frequency (tf)
## Sample             :
##     Terms
## Docs bar didnt fantast food good look lunch say servic soup
##   1    1     1       1    1    1    1     2   1      1    1
##   10   0     0       0    0    1    0     0   0      0    0
##   2    0     0       0    0    0    0     0   0      1    1
##   3    0     1       0    1    1    0     0   0      1    0
##   4    0     0       0    0    1    1     0   0      0    0
##   5    1     2       0    0    0    1     0   0      0    0
##   6    1     1       0    0    1    0     0   1      0    0
##   7    0     0       1    0    0    0     0   0      1    0
##   8    0     0       0    0    0    0     0   0      0    0
##   9    0     0       0    0    0    0     0   1      0    0