We all are aware of (not so) recent advancements in word representation, such as Word2Vec, GloVe etc. for various NLP tasks. Let's try to dig a little deeper of how they work, and why they are so helpful!
The basics, what is a Word vector?
We need a mathematical way of representing words so as to process them. We call this representation, a word vector. This representation can be as simple as a one-hot encoded vector having the size of the vocabulary. For ex, if we had 3 words in our vocabulary {man, woman, child}, we can generate word vectors in the following manner
Man : {0, 0, 1}
Woman : {0, 1, 0}
Child : {1, 0, 0}
Such an encoding cannot be used to for any meaningful comparisons, other than checking for equality. In vectors such as Word2Vec, a word is represented as a distribution over some dimensions. Each word is assigned some particular weight for each of the dimensions. Picking up the previous example, this time the vectors can be as following (assuming a 2 dimension space):
Man : {0.9, 0.1}
Woman : {0.1, 0.9}
Child : {0.5, 0.5}
The different dimensions can be thought of as latent features, which are learnt by the model. In this case, it could be that the first dimension is related to "masculinity", and the second to "femininity".
Let's talk a little more about Word2Vec
Probably the most famous among the set of word embeddings, Word2Vec word embeddings are developed by Google. It first randomly initialises embeddings. Then using raw text as input, it learns the embeddings for a word by predicting its context. These embeddings are trained using backpropagation.
These vectors capture semantic relations!
The exciting part about these learnt vectors is that they capture both semantic and syntactic relationships between different tokens. This means that vectors of similar words are also similar to one another. Not just that, word analogies can also be represented by difference and addition of word vectors!
Quoting from Mikolov et al.
An interesting application in Machine Translation
We see a very interesting application of word embeddings in machine translation by Socher et al. (2013). They show that we can translate by learning to embed words embeddings learnt from two different languages (English and Chinese) into the same space.
We can do this by first learning two different set of embeddings, Wen and Wch using their respective monolingual corpora. Additionally, we know a small curated set of translated word pairs. So while training, we can optimise for an additional constraint, that the embeddings of the word pairs from English and Chinese should be closer in the embedded space.
As expected, the embeddings of the known translation pairs ended up together, after all this was the constraint. Interestingly, words whose translation was not known while training also ended up together, hence creating a mapping between English and Chinese words having similar meaning!
The above image is a T-SNE plot of the embeddings in a 2-d space. The green points are Chinese embeddings, and yellow are English embeddings.
The basics, what is a Word vector?
We need a mathematical way of representing words so as to process them. We call this representation, a word vector. This representation can be as simple as a one-hot encoded vector having the size of the vocabulary. For ex, if we had 3 words in our vocabulary {man, woman, child}, we can generate word vectors in the following manner
Man : {0, 0, 1}
Woman : {0, 1, 0}
Child : {1, 0, 0}
Such an encoding cannot be used to for any meaningful comparisons, other than checking for equality. In vectors such as Word2Vec, a word is represented as a distribution over some dimensions. Each word is assigned some particular weight for each of the dimensions. Picking up the previous example, this time the vectors can be as following (assuming a 2 dimension space):
Man : {0.9, 0.1}
Woman : {0.1, 0.9}
Child : {0.5, 0.5}
The different dimensions can be thought of as latent features, which are learnt by the model. In this case, it could be that the first dimension is related to "masculinity", and the second to "femininity".
Let's talk a little more about Word2Vec
Probably the most famous among the set of word embeddings, Word2Vec word embeddings are developed by Google. It first randomly initialises embeddings. Then using raw text as input, it learns the embeddings for a word by predicting its context. These embeddings are trained using backpropagation.
These vectors capture semantic relations!
The exciting part about these learnt vectors is that they capture both semantic and syntactic relationships between different tokens. This means that vectors of similar words are also similar to one another. Not just that, word analogies can also be represented by difference and addition of word vectors!
Quoting from Mikolov et al.
We find that the learned word representations, in fact, capture meaningful syntactic and semantic regularities in a very simple way ... For example, if we denote the vector for word i as xi, and focus on the singular/plural relation, we observe that xapple – xapples ≈ xcar – xcars, xfamily – xfamilies ≈ xcar – xcars, and so on.
Image from Turian et al. (2010)
An interesting application in Machine Translation
We see a very interesting application of word embeddings in machine translation by Socher et al. (2013). They show that we can translate by learning to embed words embeddings learnt from two different languages (English and Chinese) into the same space.
We can do this by first learning two different set of embeddings, Wen and Wch using their respective monolingual corpora. Additionally, we know a small curated set of translated word pairs. So while training, we can optimise for an additional constraint, that the embeddings of the word pairs from English and Chinese should be closer in the embedded space.
As expected, the embeddings of the known translation pairs ended up together, after all this was the constraint. Interestingly, words whose translation was not known while training also ended up together, hence creating a mapping between English and Chinese words having similar meaning!
Image from Socher et al. 2013
The above image is a T-SNE plot of the embeddings in a 2-d space. The green points are Chinese embeddings, and yellow are English embeddings.
Making it very uncovered with regards to the we understand one small curated range of translated word of mouth twos. Which means that despite the fact that exercise, you can easily optimize for the even more only here constraint, that your embedding within the word of mouth twos with English language together with Far eastern has to be magnified during the inlay ed room or space.
ReplyDelete