How to make a model learn Word Representations in FastText

FastText Tutorial – We shall learn how to make a model learn Word Representations in FastText by training word vectors using Unsupervised Learning techniques.

Learn Word Representations in FastText

For training using machine learning, words and sentences could be represented in a more numerical and efficient way called Word Vectors. FastText provides tools to learn these word representations, that could boost accuracy numbers for text classification and such.

Input Data

Unlike supervised learning, unsupervised learning doesn’t require labelled data. So, any of the word dumps could be used as input data to train the model for learning word representations. For example, you may find many dumps from wiki at https://dumps.wikimedia.org/enwiki/latest/, if you want to try training your model with huge amount of data corpus.

For this tutorial, we shall use sample data as shown below :

But, please remember that, for any useful model to be trained, you may need lot of data corpus w.r.t your use case, at least a billion words.

Train model to learn Word Representations

To train word vectors, FastText provides two techniques. They are

  • Continuous Bag Of Words (CBOW)
  • SkipGram

Training Continuous Bag Of Words (CBOW) Model

Following is the syntax to train word vectors using CBOW model :

./fasttext cbow -input <input_file> -output <output_file>

Example

We shall use the data in a text file that is provided in the input data section, as training data.

./fasttext cbow -input wordRepTrainingData.txt -output cbowModel
cbowModel.bin is created after training.

Training SkipGram Model

Following is the syntax to train word vectors using CBOW model :

./fasttext skipgram -input <input_file> -output <output_file>

Example

We shall use the data in a text file that is provided in the input data as training data.

./fasttext skipgram -input wordRepTrainingData.txt -output cbowModel
skipGramModel.bin is created after training.

Print Word Vectors

Once the model is generated, we shall have a look on how to calculate word vectors for some input words :

Example : Calculate word vector for the word “Classification”

 

Print Sentence Vectors

We could also calculate sentence vectors using the CBOW and SkipGram models that we generated.

Example: Calculate sentence vector for the sentence “Text Classification”

 

We have printed word and sentence vectors using CBOW model. You may try with SkipGram model as a practice. All you need to do is providing skipGramModel.bin instead of cbowModel.bin in the commands.

Conclusion :

In this FastText Tutorial, we have learnt to make a model learn Word Representations in FastText using Unsupervised Learning techniques – CBOW (Continuous Bag of Words) and SkipGram. And also calculate word vectors for words and sentences.