What are N-Grams?
N-Grams are a set of co-occurring words within a given window. When computing n-grams you typically move one word forward (although you can move X words forward in more advanced scenarios)…
NLP Concepts
Get an Intuitive Understanding of Concepts Used in Natural Language Processing.
What are Stop Words?
Stop words are a set of commonly used words in a language. Examples of stop words in English are “a”, “the”, “is”, “are” and etc. Stop words are commonly used in Text Mining and Natural Language Processing (NLP) to eliminate words that are so commonly used that they carry very little useful information.
What is Inverse Document Frequency?
Inverse Document Frequency (IDF) is a weight indicating how commonly a word is used. The more frequent its usage across documents, the lower its score. The lower the score, the less important the word becomes.
What is Term Frequency?
Term frequency (TF) often used in Text Mining, NLP and Information Retrieval tells you how frequently a term occurs in a document. In the context natural language, terms correspond to words or phrases. Since every document is different in length, it is possible that a term would appear more often in longer documents than shorter ones. Thus, term frequency is often divided by the the total number of terms in the document as a way of normalization.