I decided to write this tutorial after doing some research online, only to be surprised by the amount of false information about NLP using keras and tensorflow, especially when it comes to the embedding layer. In this tutorial, i will explain exactly what the embedding layer is both from a technical perspeective and a machine learning perspective, as well as how to remove the embedding layer -should you need to- after training your model.
I will start with a simple exampel and walk you through a model that does ... yeah sentiment analysis, note that the focus here is not how to do sentiment analysis, but keras embedding layer.
Let's start with the necessary imports.
import numpy as np
import pandas as pd
import keras
from sklearn.feature_extraction.text import CountVectorizer
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from IPython.core.display import HTML, display
from keras.layers import Dense, Dropout, Embedding, LSTM, Bidirectional, \
Flatten, InputLayer
from keras import regularizers
from keras.preprocessing import sequence
from sklearn.model_selection import train_test_split
from keras.utils.np_utils import to_categorical
import re
in short, we use keras to load a prepared dataset that contains data segments from imdb consisting of both negative and positive sentiments, the only parameter we will use in this example is num words: "Top most frequent words to consider. Any less frequent word will appear as oov_char value in the sequence data"
from keras.datasets import imdb
num_words = 1000
maxlen = 400
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=num_words)
X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
print(X_train.shape)
when you have to work with the text directly yourself, keras preprocessing library is your friend, it contains word tokenizer which can be used to split texts into words, as well as sequence.pad_sequences which is used to make all your textual data of the same length.
Here's the thing, computer don't really understand textual data, least of all, neural networks, (keep in mind, when you use a String data type, it's actually represented as an array of bytes). And for this reason, we assign a unique integer ID for each uniqe word, what this index represents is pretty interesting, more on that in the following sections.
In this example, fortunately, the data is pre-sanitized, however, in real life, textual data is full of .... well ... surprises ... A good engineer always codes against surprises and unexpected cases. Keep in mind that machine learning models can run for weeks, having your code throw an exception after a few days can be a huge waste of time and resources. some basic cleaning i would recommend is:
Back to keras imdb data, the training data comes with a word index, the word index is simply a hashmap of all the word in the training set, the values mapped to each word is the aforementioned unique ID.
To get a better graps of the review data, we can use the word index to convert those numbers to words, the following piece of code does that
word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
def sequence_to_text(seq, reverse_index):
return ' '.join([reverse_index.get(i, 'N/A') for i in seq])
sequence_to_text(X_train[0], reverse_word_index)
As expected, when we convert the numbers back into their textual representation, we can see the original review, keep in mind that this is solely for debug, as far as your nueral network is concerned, you will be passing indeces (technically, the embedding is what gets passed to the neural network, more on that later).
Now we get to the core of this tutorial, the word embedding, I assume that the reader has an idea what word embedding is as a concept, if not, feel free to check this wonderful article by stanford university.
That being said, let's us load the word embeddings. For this post i will be using Stanford's Glove pretreained embeddings of dimension 50. The structure of the file is very simple, it consists of the <word> followed by the weights of that word, all seperated by a white space, we can create a word embedding dictionaty by simply splitting each line, using the first split as a key and the rest as the values (word embeddings). Finally, as recommended by a very talented friend of mine (Andre Cianflone), it's advisable to create a random weight for all the unknow words. Unknow words may not have an embedding for a variety of reasons: misspelling, the word is not in English, etc ...
embeddings_index = {}
embedding_dim = 50
f = open('sample_data/glove.6B.50d.txt', encoding='utf8')
for line in f:
values = line.split()
word = values[0]
coefs = np.asarray(values[1:], dtype='float32')
embeddings_index[word] = coefs
f.close()
# Create a weight for all unknow words
embeddings_index['<unk>'] = np.random.uniform(low=-0.0001, high=0.0001, size=embedding_dim)
And now we create the embedding matrix, the purpose of the embedding matrix is to map word indexes to embeddings weights.To create the matrix, we loop through our word index, and assign the particular weight from the embedding index to the matrix. We should notice that In which case we assign it a random vector
embedding_matrix = np.zeros((len(word_index) + 1, embedding_dim))
for word, i in word_index.items():
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
embedding_matrix[i] = embedding_vector
else:
embedding_matrix[i] = embeddings_index.get('<unk>')
i += 1
print(embedding_matrix.shape)
print(len(word_index) + 1)
print(embedding_layer.input_dim)
Finally, we create an Embedding layer from the embedding matrix, the Embedding layer acts as a look-up table for the neural network in order to convert a word index to a vector
embedding_layer = Embedding(len(word_index) + 1,
50,
weights=[embedding_matrix],
input_length=X_train.shape[1],
trainable=False)
If you're a curious reader, you're probably wondering why we're creating all those variables to use word embedding? Can't we just map words to vectors directly without having an Embedding layer?
Well, the answer is that you actually can, but as all things, it's a bit more complicated than that... If we skip the embedding, then our X_train vector would be a list of vectors each of size EMBEDDING_DIMENTION (50 in this case, but it could be as high as 300). Which means that the size of our X_train vector would be EMBEDDING_DIMENTION times higher than if we used an Embedding layer.
This feature becomes particularly important when using GPU for training, in which case, rather than feeding the model with a big fat array of vectors, we feed a list of integers and the model can look-up whichever vectors it needs while training without plotting the memory with the full sized vectors.
We're all set to create the model, here we go
model = Sequential()
model.add(embedding_layer)
model.add(Bidirectional(LSTM(units=10, return_sequences=True,
dropout=0.5, kernel_regularizer=regularizers.l2(0.001),
activity_regularizer=regularizers.l2(0.001))))
model.add(Bidirectional(LSTM(units=10,
dropout=0.5, kernel_regularizer=regularizers.l2(0.001),
activity_regularizer=regularizers.l2(0.001))))
model.add(Dense(1,activation='sigmoid'))
model.compile(loss = 'binary_crossentropy', optimizer='adam',metrics = ['accuracy'])
model.summary()
print(X_train.shape,y_train.shape)
print(X_test.shape,y_test.shape)
batch_size = 32
model.fit(X_train, y_train, epochs = 20, batch_size=batch_size, verbose = 2)
Of course, when you train a model for actual usage, you should aim for a higher accuracy and you definitely should have test set. I will skip that since this post is about word embedding not how to train your dragon... I mean model.
And now to the fun part, we get to actually use the model to predict the sentiment of a tweet. I engineered those two examples carefuly to make a point, the word "waggish" is more or less a synonym for "fun", but it does not appear in our training data, and thus, it does not have a value in the tokenizer's word_index.
tweets = np.array([
"Me reading my family comments about how great the was real fun, i enjoyed the debate",
"Me reading my family comments about how great the was waggish, i enjoyed the debate"
])
Let's take a look at what the tokenized sentences looks like
tokenizer.texts_to_sequences(tweets)
If you pay a close looks at the arrays above, you'll notice that one entry is missing, which is the word "waggish", kera's texts_to_sequences fails silently when it sees an unknown word and it omits it from the resulting array.
On that note, let us try predicting the sentiment of the above tweets anyway, note that whatever padding/cleaning/pre-processing done to the training data needs to be repeated to the prediction data, this applies to any machine learning model not just this one.
X_predict = tokenizer.texts_to_sequences(tweets)
X_predict = pad_sequences(X_predict, maxlen=max_sent_len)
The data is now ready to be passed to the predict function, the result will be a probability of either positive or negative.
preds = model.predict(X_predict)
print(preds)
Well, the model did guess the correct sentiment for both sentences, but it seems less confident about the second one which should not be the case since both senetnces are pretty much identical. The reason for that, as stated earlier is that the model actually sees the second tweet as "Me reading my family comments about how great the was <nothing>, i enjoyed the debate"
Luckily, there is a way to introduce new words to the model, which is the main reason why we used word embeddings in the first place. The logic behind this is simple, if we place the word "fun" with "waggish", we should ideally get a similar vector of embeddings for each words, the problem we have is that the word "waggish" was not a part of the vocabulary we trained the model to use. However, we could simply replicate the trained model, replacing the embedding layer with an input layer that directly takes the word vectors as an input. Note that it does slightly affect the performance of the model especially if we are running our predictions on a huge dataset, I would recommned you try to fix your vocab beforehand if you can. If not, the code below shows how you can use your trained model (or technically a copy of it) to predict sentences that contains unseen words, the code is very simple, the only trick is knowing the shape of your input model, you can follow that in the comments.
newModel = Sequential()
#Add an input layer of dimention: max_sentence_length, embedding_dimention
#this means that our input would consist of vectors of length max_sent_len, each entry is of size embedding_dimention
#(i.e: a list of word vectors, each of size embedding_dimention)
newModel.add(InputLayer(input_shape=(max_sent_len, 50), dtype='float32'))
#and then, we simply copy the layers over from the old model
newModel.add(model.layers[1])
newModel.add(model.layers[2])
newModel.add(model.layers[3])
newModel.summary()
With the embedding layer gone, we have to basically do what it did manually, it's very straigh forward, we start by tokenizing the sentence
tweets_tok = [keras.preprocessing.text.text_to_word_sequence(x) for x in tweets]
print(tweets_tok)
The next part is where the magic happens, we need to loop through our tokenized tweets, convert each word to its corresponding vector, we also need to prepend (or append, depending how you configured your tokenizer while training) with a zero vector.
Each entry of the resulting vector should be of shape (max_sent_length, embedding_dim) or (200, 50) in our case, each one of those 200 vectors represents a word vector or a zero vector in the case of padding.
X_predict_newmodel = []
# print(embeddings_index)
# import pdb; pdb.set_trace()
for t in tweets_tok:
X_sent = None
for i, word in enumerate(t):
# print(word)
if word in embeddings_index:
if i == 0:
X_sent = embeddings_index.get(word)
else:
X_sent = np.vstack((X_sent, embeddings_index.get(word)))
else:
if i == 0:
X_sent = np.random.uniform(low=-0.0001, high=0.0001, size=50)
else:
X_sent = np.vstack( (X_sent, np.random.uniform(low=-0.0001, high=0.0001, size=50)))
for j in range(0, max_sent_len - X_sent.shape[0]):
X_sent = np.vstack((np.zeros(50), X_sent))
X_predict_newmodel.append(X_sent)
X_predict_newmodel = np.array(X_predict_newmodel)
print(X_predict_newmodel.shape)
And we're all set, let's predict the sentiment for these non-human-readable bunch of vectors
print(newModel.predict(X_predict_newmodel))
Noticed anything interesting? How about two things?
First, the probabilities of the first tweet are identical to our earlier prediction using the embedding layer, in fact, this is a good way to debug your model, this should be the case for any sentence consisting solely of words that the model saw while training (and thus, are part of the word_index). The only exception to that is when a sentence has words that do not have corresponding embeddings and get assigned random values, when debugging try to engineer an example that consists of words that are both seen in training AND have embeddings.
Second, and more importantly, the model is now more confident in the sentiment of the second tweet, it does make sense as we input the full sentence along with the word "waggish". What happens in this case , as ooposed to the previous prediction, where we skipped "waggish", is that the same neurons that get activated when seeing the vector for the word "good" will also be activated for "waggish" resulting in a more acurate prediction.