Home

Maximum Likelihood Estimation

2019-01-28T00:00:00+00:00

Introduction

This post will build on a previous one which covered some techniques for convergence of linear models (found here). As a brief review, statistical inference is the process of building a model which reflects the patterns and properties of a given dataset. In the case where the functional form is assumed to be linear, the process conveniently simplifies to computing the set of model parameters which minimize model errors.

For the general case, we need a way to approximate the functional form of the data before computing the model parameters and a common technique used in practice is known as Maximum Likelihood Estimation. We will consider an application of MLE with an implementation of Logistic Regression from scratch.

MLE - Background

The terms probability and likelihood are semantically similar but there is a technical difference in application. First, probability is a measure of outcome where likelihood is a measure of the model. The likelihood function estimates the set of parameter values which maximize the certainty of the observed probabilities. In this way, likelihood quantifies trust in the model, given the data. The formal notation for the likelihood is as follows:

$\begin{equation} L(\theta;x^{n}) = p(x^{n};\theta) \end{equation}$

L is the likelihood function. It is the joint distribution of the observed probabilities and is used to estimate the best $\theta$ consistent with these probabilities. It also has some nice statistical properties which allow for easier computation. Assuming the data is a random sample from some probability distribution, we can represent the joint probability distribution as the product of univariate distributions (which are all the same!).

$\begin{equation} L(\theta;x^{n}) = \prod_{i=1}^{n}p(x_{i};\theta) \end{equation}$

We can simplify things further by working with the log-likelihood. Since the log function is nondecreasing, the maximum likelihood estimate for the likelihood will be the same for the log-likelihood. This will allow us to convert the product of univariates to a sum of univariates which helps for derivatives. For information on log properties see this link

$\begin{equation} l(\theta;x_{i}) = logL(\theta;x_{i}) \end{equation}$

From here, we can use good ol’ calculus to partition l( $\theta$ ; $x_{i}$ ) for the $\theta$ most consistent with our data. Alright, this stuff is pretty abstract. Let’s put it to the test with an implementation of Logistic Regression.

Logistic Regression - Motivation

Regression models typically estimate the response of continuous variables from a set of fixed variables for prediction and/or inference. Said another way, prediction results are formulated as the conditional expectation of a continuous variable, where we can measure the average value of the input-output relationship. But what if our response is categorical and not continuous? It turns out that linear models can also be used in this instance as well.

Suppose our task is to project the default status of some student loans. It would seem reasonable to assume some randomness in the result of this binary outcome due to the difficulty of projecting 4+ years into a student’s future (among other considerations!). As a result, we will need an associated probability for our projections to account for this noise. Conveniently, for binary outcomes, Pr(Y=1) = E[y] (ratio of 1’s for the response) and Pr(Y=0) = 1-Pr(Y=1).

This means we can use our linear model to project the probability of default and use some threshold rule for loan classification. The issue, however, is the results of our linear model are unbounded and will require a transformation to express the results as probabilities. For this purpose, we can take the log odds of our result, which is the log of the ratio of success/failure. Thus, our model can now be represented in the following way:

$\begin{equation} log\frac{p(x)}{1-p(x)} = \beta_{0}+X\beta \end{equation}$

This is known as the logistic regression model. Solving for (p) gives the following result:

$\begin{equation} p(x) = \frac{1}{1+e^{-(\beta_{0}+X\beta)}} \end{equation}$

We can take the exponential of both sides to drop the natural log and use a little algebra to solve for (p). The result is known as the logit function. We can use the logit to represent the results of our linear model as probabilities bound by zero and one. We’ll then use these probabilities to classify results and evaluate performance. Here is a visual of the logit function.

Here, the data is evenly distributed around zero so negative inputs (t) produce probabilities less than 0.5 and positive inputs have probabilities greater than 0.5. The Logistic Regression algorithm for scikit-learn uses 0.5 as a baseline for classification. That is, transformed outputs greater than 0.5 are classified with a label of Y=1 and Y=0 otherwise. Hence, (p) is the probability that Y=1. Finally, we need to optimize our model parameters consistent with the observed probabilities.

Likelihood Function for Logistic regression

Given the iid assumption that the data is a random sample, the likelihood function for Logistic Regression is a product of Bernoulli distributions.

$\begin{equation} L(\beta_{0},\beta) = \prod_{i=1}^{n}p(x_{i})^{y_{i}}(1-p(x_{i}))^{1-y_{i}} \end{equation}$

$x_{i}$ is our feature vector, $p(x_{i})$ is the logit, and $y_{i}$ is the observed class for the associated probability of (p). The log-likelihood can be represented in the following way:

$\begin{equation} l(\beta_{0},\beta) = \sum_{i=1}^{n}y_{i}logp(x_{i})+(1-y_{i})log1-p(x_{i}) \end{equation}$

We can distribute and combine terms using log properties:

$\begin{equation} = \sum_{i=1}^{n}log1-p(x_{i})+\sum_{i=1}^{n}y_{i}log\frac{p(x_{i})}{1-p(x_{i})} \end{equation}$

Using the division property of logs, the first term can be written as (1/logit) = $\frac{1}{\frac{1}{1+e^{-\beta_{0}+x_{i}\beta}}}$ , which we can simplify a bit further for easier derivatives. The second term is our Logistic Regression model.

$\begin{equation} l(\beta_{0},\beta) = \sum_{i=1}^{n}-log1+e^{\beta_{0}+x_{i}\beta}+\sum_{i=1}^{n}y_{i}(\beta_{0}+x_{i}\beta) \end{equation}$

Now, we need to differentiate the log-likelihood for each parameter ( $\beta_{j}$ ) to find the maximum likelihood estimates.

$\begin{equation} \frac{\partial}{\partial\beta_{j}} = -\sum_{i=1}^{n}\frac{e^{\beta_{0}+x_{i}\beta}}{1+e^{\beta_{0}+x_{i}\beta}}x_{ij}+\sum_{i=1}^{n}y_{i}x_{ij} \end{equation}$

After partitioning the log-likelihood for $\beta_{j}$ , we’re left with the logit of the jth feature for each observation and the label of the jth feature for each observation. Combining terms gives the following function for the gradients:

$\begin{equation} = \sum_{i=1}^{n}(y_{i}-p(x_{i}))x_{ij} \end{equation}$

We will use this function to iteratively update and evaluate our feature set, where performance is the difference between the label and estimated probability given the current set of parameters.

Model implementation

Now, let’s generate some data and train a model!

np.random.seed(88)
m1 = [0,0]
m2 = [2,5]
n = 3000

x1 = np.random.normal(m1,size=[n,2])
x2 = np.random.normal(m2,size=[n,2])

features = np.vstack((x1,x2)).astype(np.float32)
labels = np.hstack((np.zeros(n),np.ones(n)))

Here, we simulate some data and ensure it’s linearly separable from different sample means. Then we combine the data to form our feature set and attach the labels. A plot of the data should clearly distinguish between the two classes. Let’s have a look!

plt.style.use('ggplot')
plt.figure(figsize=(10,8))
plt.scatter(features[:,0],features[:,1],c=labels)
plt.show()

As you can see, we can easily separate this data with a linear decision boundary. Right on! Now that we have our data, let’s build the model. First, we’ll define the logit function, append the model intercept to our feature set, and initialize random values for the model parameters.

# define logit function
def logit(output):
    return 1/(1+np.exp(-output))

# account for model intercept
x0_features = np.c_[np.ones((len(features),1)),features]

# randomly initialize parameters
theta = np.random.randn(3,1)

We’re now ready to build the model. We’ll also need an adjustment rate for the gradient updates and the number of iterations for training. Ideally, training would continue until model error is sufficiently low, but 100,000 iterations will get us close enough. For each iteration, we’ll compute the output from the linear model, transform it to a probability, and compute the gradients from the differentiated log-likelihood function. Training should produce a model which correctly classifies both features with high accuracy.

# Logistic Regression from scratch
adj_rate = .01
iterations = 100000

for iteration in range(iterations):
    #transformation
    linear_output = np.dot(x0_features,theta)
    probs = logit(linear_output)

    #update
    performance = labels.reshape(6000,1) - probs
    ll = np.dot(x0_features.T,performance)
    theta = theta + adj_rate*ll

Alright, so we have a linear model which uses a transformation function and Maximum Likelihood Estimation for classification. Let’s compare our model parameters for Logistic Regression with Scikit-learn’s. The hyperparameter (C) accounts for regularization which we didn’t account for in our model, but could be a topic for a future post!

Our model’s parameters are, in fact, very close to the out-of-the-box version from sklearn. Very cool! To consider performance, let’s round the probabilities generated from our model and plot the conditional result of comparison to the labels. So, if the probability rounds to a one and the corresponding label is also a one then the model was correct in its prediction. Correct predictions are the yellow points in the following plot:

Our model performs very well with over 99% accuracy!

Convergence of Linear Models

2019-01-18T00:00:00+00:00

Introduction

This post will cover the techniques and concepts used for training linear models. While most of the information presented here has already been covered elsewhere with great detail, I hope to present it in such a way that is relatable to anyone interested in data science. In doing so, I will also test the depth of my own knowledge and improve my written communication as a result. For additional information, a list of references will be provided at the end of this post.

Background

Supervised learning is a classic domain of statistics which involves a set of features and a response variable. A supervised learning model seeks to capture the full relationship between the features and the response for prediction/inference. If this relationship exists, then there must be some function which relates the systematic information of the features to the response. The modeling process amounts to approximating this function and evaluating the results. The general form of the function can be written in the following way:

$\begin{equation} Y = f(X) + \epsilon \end{equation}$

In this equation, $f$ is our model and $\epsilon$ represents the variation of $y$ not captured by our model. Linear models are named as such because they associate a linear shape for $f$ . Here is the general form of a linear model:

$\begin{equation} f(X) = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2} +...+\beta_{n}X_{n} \end{equation}$

$\beta_{0}$ is a constant term representing the bias for model training. Every other parameter has an associated feature ( $X_{1:n}$ ) representing the measured effect of $X_{i}$ on $Y$ . Given some data, we can use this model to make predictions as a weighted sum of the input features. These predictions are evaluated using a performance measure, typically RMSE (Root Mean Squared Error). Here, RMSE is represented using matrix notation where $\theta$ are the model parameters and $X$ is our feature set.

$\begin{equation} RMSE = ((1/n)\sum_{i=1}^{n}[y_{i}-\theta^{T}X_{i}]^{2})^{1/2} \end{equation}$

For each instance, we take the squared difference between our prediction and the label, add them up, and normalize the result. It follows that we need to find the best combination of parameters which minimize model errors ( $y_{i}$ - f(X)). There are two approaches to accomplish this task.

The Closed Form Solution

In this section, I will go through the steps to find model parameters for a single feature. Applications with multiple features are conceptually the same process but use matrix notation for convenience. In either case, the goal is a direct solution of optimal parameters instead of an iterated approach (more on this later). Lets get started!

Our goal is to minimize our performance measure (RMSE) with respect to the model parameters ( $\beta_{0}$ and $\beta_{1}$ ). We can drop the square root and the (1/m) constant since they have no effect on the end result. This simplifies our function to the squared-sum of model errors (SE):

$\begin{equation} SE = \sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{i})^{2} \end{equation}$

We can obtain a direct solution through partitioning our cost function with respect to each model parameter. First, we’ll take the partial derivative of SE for $\beta_{0}$ .

$\begin{equation} \beta_{0}^{'} = -2\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{i}) = 0 \\ = \sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{i}) = 0 \\ \sum_{i=1}^{n}\beta_{0} = \sum_{i=1}^{n}y_{i} -\beta_{1}\sum_{i=1}^{n}x_{i} \\ n\beta_{0} = \sum_{i=1}^{n}y_{i} - \beta_{1}\sum_{i=1}^{n}x_{i} \\ \beta_{0} = (1/n)\sum_{i=1}^{n}y_{i} - \beta_{1}(1/n)\sum_{i=1}^{n}x_{i} \\ \beta_{0} = \bar{y} - \beta_{1}\bar{X} \end{equation}$

After dropping the -2 constant, we distribute the summation operator and rewrite in terms of $\beta_{0}$ . $\beta_{0}$ is the model intercept, so a summation over a constant also produces a constant ( $n\beta_{0}$ ).

We can now use this result with the partial derivative of SE for $\beta_{1}$ to derive the direct solution.

$\begin{equation} \beta_{1}^{'} = -2\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{i})(x_{i}) = 0 \\ = \sum_{i=1}^{n}x_{i}(y_{i}-\beta_{0}-\beta_{1}x_{i}) = 0 \\ =\sum_{i=1}^{n}x_{i}y_{i}-\beta_{0}\sum_{i=1}^{n}x_{i}-\beta_{1}\sum_{i=1}^{n}(x_{i})^{2} = 0 \end{equation}$

The result of the partial derivative for $\beta_{1}$ is still in terms of $\beta_{0}$ , so we need to do a bit more work. We can use the partial for $\beta_{0}$ as a function of $\beta_{1}$ to substitute and solve for $\beta_{1}$ using some algebra. The end result is a little messy but is a direct solution nonetheless!

$\begin{equation} \beta_{1} = \frac{\sum_{i=1}^{n}x_{i}y_{i}-(1/n)\sum_{i=1}^{n}x_{i}\sum_{i=1}^{n}y_{i}}{\sum_{i=1}^{n}(x_{i})^{2} - (1/n)(\sum_{i=1}^{n}x_{i})^{2}} \end{equation}$ $\begin{equation} \beta_{1}=cov(x,y)/var(x) \end{equation}$

And here is the much cleaner solution in matrix notation for the general case:

$\begin{equation} \theta = (X^{T} \cdot{X})^{-1} \cdot{X^{T}} \cdot{y} \end{equation}$

Implementation

Now that we have a conceptual understanding of the linear regression algorithm, let’s build it! First, we’ll generate some data and plot it. We should expect to see a linear relationship between changes in $x_{1}$ and $y$ .

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(11)

X1 = 3*np.random.rand(100,1)
y = 5 + 4*X1 + np.random.randn(100,1)

plt.plot(X1,y,"c.")
plt.xlabel("$x_1$",fontsize=15)
plt.ylabel("$y$",fontsize=15)
plt.axis([0,3,0,20])
plt.show()

As expected, the data reflects a positive association of $x_{1}$ with $y$ . $x_{1}$ is an array of 100 data points drawn from the uniform distribution ([0,1)) and $y$ is also an array of 100 data points dependent on $x_{1}$ plus some noise. The noise is randomly generated from the normal distribution, consistent with model assumptions for errors (mean = 0).

Now let’s optimize for our parameters. We’ll account for our model’s intercept by appending a column of one’s to the $x_{1}$ data. In this way, the intercept is implemented the same as any feature but with an input value of one for every instance. Next, we can implement the direct solution with one line of code and numpy! Here are the results:

X0_X1 = np.c_[np.ones((100,1)),X1]
theta = np.linalg.inv(X0_X1.T.dot(X0_X1)).dot(X0_X1.T).dot(y)

We can visualize our linear model as a function of $x_{1}$ by connecting the prediction results from its fringe values (constant slope). In this case, our data ranges from [0,3] so we’ll compute predictions for these inputs and plot the fitness of our model.

X_fringe = np.array([[0],[3]])
X0_X1_fringe = np.c_[np.ones((2,1)),X_fringe]
y_fringe = X0_X1_fringe.dot(theta)

plt.plot(X_fringe,y_fringe,"b-")
plt.plot(X1,y,"c.")
plt.xlabel("$x_1$",fontsize=15)
plt.ylabel("$y$",fontsize=15)
plt.axis([0,3,0,20])
plt.show()

Looking good! Finally, let’s check our implementation vs a packaged version from Scikit-Learn. Our model parameters should be the same as theirs if we didn’t make any mistakes (direct solution).

from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(X1,y)

print("sklearn B0: ", lr.intercept_)
print("sklearn B1: ", lr.coef_[0])

Gradient Descent

Consider that the Normal Equation requires computation and inversion of the matrix $(X^{'}X)^{-1}$ , which can become expensive and slow as model dimensionality increases. Gradient Descent is another approach to finding the optimal set of model parameters and scales much better with the number of features. It starts with a random initialization of parameter values, and given these values, computes a number of iterative adjustments based on the local optima and the learning rate.

In other words, since our performance measure (MSE) is a function of our model parameters ( $\theta$ ), gradient descent starts with a random set of parameters and incrementally improves the recent set with each iteration. Model training converges to the best set of parameter values when the local optima approximate the global optima with a tolerable level of error.

The learning rate is the pace of adjustment for the parameters based on the local optima and it’s actually an important part of the algorithm. If it’s set too small then training can take an unnecessarily long amount of time, but if it’s too large then it’s possible to miss the set of parameters which minimize model errors. In practice, a learning rate of 0.1 is used as a default.

Partitioning our performance function with respect to a given parameter ( $\theta_{j}$ ) is equal to the following equation:

$\begin{equation} \frac{\partial}{\partial\theta_{j}}SE(\theta) = \frac{2}{n}\sum_{i=1}^{n}(\theta^{T} \cdot{x^{(i)}}-y^{(i)})x_{j}^{(i)} \end{equation}$

Given the current values of $\theta$ , Gradient Descent computes the partial derivatives for each feature and adjusts $\theta$ based on these local optima and the learning rate. The cumulation of these partial derivatives can be represented as a vector and computed in the following way:

$\begin{equation} \nabla_{\theta}SE(\theta) = \frac{2}{n}X^{T} \cdot{(X \cdot{\theta -y})} \end{equation}$

Implementation

Notice that the function for the gradient vector uses the entire dataset for training. This is called Batch Gradient Descent because it uses all of the training data during each iteration of computing the gradients. As a result, training with Batch scales poorly with larger datasets.

The implementation is fairly straightforward.

adj_rate = 0.1
iterations = 1000
n = 100
theta = np.random.randn(2,1)

for iteration in range(iterations):
    gradients = 2/n * X0_X1.T.dot(X0_X1.dot(theta) - y)
    theta = theta - adj_rate*gradients

print("model intercept: ",theta[0])
print("x1 parameter:    ",theta[1])

We randomly initialize $\theta$ from the normal distribution (mean = 0) to ensure no directional bias for the gradients, on average. Then we compute the gradient vector for each iteration and adjust our $\theta$ according to the learning rate. As expected, the results are identical for Batch Gradient Descent.

Let’s have a look at the first ten adjustments of $\theta$ using Batch Gradient Descent. The algorithm appears to converge quickly for this dataset. The dashed red line is the random initialization of parameters.

References

Hands-On Machine Learning with Scikit-Learn and Tensorflow by Aurelien Geron (O’Reilly)
An Introduction to Statistical Learning by Gareth James, Deniela Witten, Trevor Hastie, and Robert Tibshiran (Springer)
An Intuitive Introduction to Gradient Descent by Thalles Silva https://towardsdatascience.com/machine-learning-101-an-intuitive-introduction-to-gradient-descent-366b77b52645
Derivation proof https://math.stackexchange.com/questions/716826/derivation-of-simple-linear-regression-parameters

Building a Spam Classifier from Scratch

2019-01-08T00:00:00+00:00

Introduction

Spam detection is a classic exercise that presents a solid introduction to machine learning in practice. In this post, we will use some basic tools of natural language processing for sentiment analysis of an SMS dataset from Kaggle. These techniques should allow us to build a system which captures the semantics of each text message and utilize supervised learning algorithms for performance evaluation.

Background

Given our set of text messages, the task is to determine whether each message is spam or not. Our data is a combination of overlapping words/characters/symbols from each message. Our system should learn from the patterns and frequencies provided by the data to establish a baseline for predictions. This can be broken down into several stages.

First, we need to convert our messages into an exhaustive vocabulary list. Next, we will utilize some processing techniques to control for fixed-effects such as part of speech, punctuation, capital letters, etc. This should produce a better signal for our model by reducing the amount of noisy terms. Finally, we will convert our list into a vector where each character sequence will have a corresponding index value.

If a message contains a given character sequence from our list, then the corresponding vector index will contain a value of one and zero otherwise. We will effectively transform each message into a vector of ones and zeros indexed by character sequence. All together, the cumulation of these vectors will represent our feature matrix. We will feed this matrix into our algorithms and evaluate its performance. Let’s get started!

Stage One - Building the List

Alright, let’s import our dataset and take care of some cleaning. We’ll drop a few unnecessary columns, rename the ones we’re keeping, and map ‘ham’ and ‘spam’ entries from our target to zeros and ones. Our final preprocessing step is to convert each text message into a list of character sequences, which is also known as tokenizing. All of this work can be done using two python libraries: pandas and nltk. The nltk package provides functionality specific to the natural language processing domain, while pandas is a general platform for data analysis. Let’s have a look at a few tokenized messages.

As seen from the image, each message is now a list of character sequences. Here is the small function used for the preprocessing (all code can be found on my GitHub):

def process_spam_data(filepath):
  data = pd.read_csv(filepath, encoding="latin-1")
  data = data[['v1','v2']]
  data = data.rename(columns={'v1':'label','v2':'text'})
  data['label'].replace(['ham','spam'],[0,1], inplace=True)
  data['token_text'] = data['text'].apply(nltk.word_tokenize)
  return data

Stage Two - NLTK Processing

Now we are ready to start building our feature matrix where each row will represent a text message from our training set with ones in columns corresponding to the character sequences in the given message (zeros otherwise). Our goal is to capture as much semantic meaning as possible while maintaining a scalable process with efficient use of computational resources. This dataset contains almost 12,000 unique words and not all return the same signal. Training can quickly become very costly, so we will use some processing techniques to reduce the lower value content.

To start, the NLTK package provides a list of stop words which are a compilation of the most common words observed in text. Stop words provide a lower signal since they are generally observed at a higher frequency. In other words, their semantic meaning changes very little within context. As a result, these words can be safely removed from our list. By simply removing these common words, our list’s length was reduced to about 9800. This is an effective decrease of ~17%. Not bad! Let’s have a look at the top ten stop words removed from our vocabulary list:

It’s not too surprising to see some pronouns, prepositions, and conjunctions at the top of the list as these parts of speech provide the lowest value in terms of signal. I also added punctuation to the stop-words list (effectively removing punctuation from the dataset).

Next in the process is normalizing our list of terms to their base form. A word can have many representations depending on its part of speech, so we want to reduce these different word endings to the base form, or lemma. NLTK has a class for lemmatization which requires the word representation and its part of speech. I wrote a function to map the nltk tags to the part of speech parameter for the lemmatization class:

def part_of_speech(tag):
    pos = ""
    if (word[1] == 'VB') | (word[1] == 'VBD') | (word[1] == 'VBG') | (word[1] == 'VBN') | (word[1] == 'VBP') | (word[1] == 'VBZ'):
        pos = 'v'
    elif (word[1] == 'RB') | (word[1] == 'RBR') | (word[1] == 'RBS') | (word[1] == 'WRB'):
        pos = 'r'
    elif (word[1] == 'JJ') | (word[1] == 'JJR') | (word[1] == 'JJS'):
        pos = 'a'
    else:
        pos = 'n'
    return pos

This function isn’t the most elegant but it gets the job done. From here, we’ll use our word list to obtain the part of speech tags, use the above function to map the tags to the parameter value for the lemmatization class and finally normalize our word list.

from nltk.stem import WordNetLemmatizer

words = []
for i in data['token_text']:
    words += i
print(len(set(words)))
words = [w.lower() for w in words if w not in stop_words]
tagged_words = nltk.pos_tag(words)

lemmatizer = WordNetLemmatizer()

def lemmatize_words(tagged_words):
    lemmatized_words = []
    for word in tagged_words:
        lemmatized_words.append(lemmatizer.lemmatize(word[0], pos=part_of_speech(word[1])))
    lemma_counts = Counter(lemmatized_words)
    return lemma_counts, lemmatized_words

The end result is a dictionary containing the base representation of each word and its associated frequency. We also get a list of these words for convenience (used later). As a result, the unique set of words was reduced from 9803 to 9286. These basic techniques allowed us to discard ~25% of the low yield content. Pretty sweet! Now what’s left is indexing our vocabulary list and vectorizing our dataset.

Stage 3 - Vectorize the Corpus

The vocabulary list will be indexed from most common to least common. This means the most common word’s index is the 2nd column (index=1), the next most common word’s index is the 3rd column, etc. The first column at index = 0 of the matrix represents the frequency of words not found in the vocabulary due to processing.

def build_common_vocabulary(lemma_counts):
    vocabulary = {}
    index = 1

    for word in lemma_counts.most_common():
        vocabulary[word[0]] = index
        index +=1
    return vocabulary

Let’s now use our lemmatized list to vectorize each message. For each message in our dataset, if each word is in our processed list then we’ll track its frequency in the given message. Next, we’ll index these word counts for each message based on the compiled vocabulary.

Here is the function and resulting output from converting each message to word counts based on our lemmatized list:

text_data = data['token_text']

msg_counts = []
for msg in text_data:
    text = []
    for word in msg:
        word = word.lower()
        if word in lemmatized_words:
            text.append(word)
    word_counts = Counter(text)
    msg_counts.append(word_counts)
print(data['text'].head())
print(msg_counts[:5])

We’re ready to vectorize the text messages using the word counts and vocabulary indices. We’ll use a sparse representation since the matrix will be mostly zeros.

from scipy.sparse import csr_matrix

def vectorize_corpus(msg_counts=msg_counts, vocabulary=vocabulary,
vocab_size=9286):
    rows = []
    cols = []
    data = []

    for row, counter in enumerate(msg_counts):
        for word, count in counter.items():
            rows.append(row)                                     
            cols.append(vocabulary.get(word,0))                  
            data.append(count)                                  
    X = csr_matrix((data, (rows,cols)), shape=(len(msg_counts), vocab_size+1))
    return X

The format of the sparse matrix requires the row and column index for each data entry. These entries are the word counts for a given message. The row index corresponds to particular message and the column index corresponds to a particular word from our global list. If a word is not found in the global list, its count is indexed to column zero. The vectorize_corpus function iterates over the messages, obtains the indices for its word counts, and maps this data to the feature matrix.

Performance Evaluation

The feature matrix was fed to several supervised learning algorithms for training. Performance was generally strong across the board with f-scores ranging between 0.85 and 0.92 on the training set! Parameter tuning had a strong effect on results, especially for the logistic regression algorithm. High precision and low recall indicated a sub-optimal threshold for its decision function. An increase in performance of more than 50% resulted from simply adjusting its threshold value (Please see the GitHub repository for code).

Here are the confusion matrices and final results for testing: