Language model

A unigram model can be treated as the combination of several one-state finite automata.[4] It splits the probabilities of different terms in a context, e.g. from

${\displaystyle P(t_{1}t_{2}t_{3})=P(t_{1})P(t_{2}\mid t_{1})P(t_{3}\mid t_{1}t_{2})}$

to

${\displaystyle P_{\text{uni}}(t_{1}t_{2}t_{3})=P(t_{1})P(t_{2})P(t_{3}).}$

In this model, the probability of each word only depends on that word's own probability in the document, so we only have one-state finite automata as units. The automaton itself has a probability distribution over the entire vocabulary of the model, summing to 1. The following is an illustration of a unigram model of a document.

${\displaystyle \sum _{\text{term in doc}}P({\text{term}})=1}$

The probability generated for a specific query is calculated as

${\displaystyle P({\text{query}})=\prod _{\text{term in query}}P({\text{term}})}$

Different documents have unigram models, with different hit probabilities of words in it. The probability distributions from different documents are used to generate hit probabilities for each query. Documents can be ranked for a query according to the probabilities. Example of unigram models of two documents:

In information retrieval contexts, unigram language models are often smoothed to avoid instances where P(term) = 0. A common approach is to generate a maximum-likelihood model for the entire collection and linearly interpolate the collection model with a maximum-likelihood model for each document to smooth the model.[5]

In an n-gram model, the probability ${\displaystyle P(w_{1},\ldots ,w_{m})}$ of observing the sentence ${\displaystyle w_{1},\ldots ,w_{m}}$ is approximated as

${\displaystyle P(w_{1},\ldots ,w_{m})=\prod _{i=1}^{m}P(w_{i}\mid w_{1},\ldots ,w_{i-1})\approx \prod _{i=1}^{m}P(w_{i}\mid w_{i-(n-1)},\ldots ,w_{i-1})}$

It is assumed that the probability of observing the i^th word w_i in the context history of the preceding i − 1 words can be approximated by the probability of observing it in the shortened context history of the preceding n − 1 words (n^th order Markov property).

The conditional probability can be calculated from n-gram model frequency counts:

${\displaystyle P(w_{i}\mid w_{i-(n-1)},\ldots ,w_{i-1})={\frac {\mathrm {count} (w_{i-(n-1)},\ldots ,w_{i-1},w_{i})}{\mathrm {count} (w_{i-(n-1)},\ldots ,w_{i-1})}}}$

The terms bigram and trigram language models denote n-gram models with n = 2 and n = 3, respectively.[6]

Typically, the n-gram model probabilities are not derived directly from frequency counts, because models derived this way have severe problems when confronted with any n-grams that have not been explicitly seen before. Instead, some form of smoothing is necessary, assigning some of the total probability mass to unseen words or n-grams. Various methods are used, from simple "add-one" smoothing (assign a count of 1 to unseen n-grams, as an uninformative prior) to more sophisticated models, such as Good-Turing discounting or back-off models.

Bidirectional representations condition on both pre- and post- context (e.g., words) in all layers.[7]

In a bigram (n = 2) language model, the probability of the sentence I saw the red house is approximated as

${\displaystyle P({\text{I, saw, the, red, house}})\approx P({\text{I}}\mid \langle s\rangle )P({\text{saw}}\mid {\text{I}})P({\text{the}}\mid {\text{saw}})P({\text{red}}\mid {\text{the}})P({\text{house}}\mid {\text{red}})P(\langle /s\rangle \mid {\text{house}})}$

whereas in a trigram (n = 3) language model, the approximation is

${\displaystyle P({\text{I, saw, the, red, house}})\approx P({\text{I}}\mid \langle s\rangle ,\langle s\rangle )P({\text{saw}}\mid \langle s\rangle ,I)P({\text{the}}\mid {\text{I, saw}})P({\text{red}}\mid {\text{saw, the}})P({\text{house}}\mid {\text{the, red}})P(\langle /s\rangle \mid {\text{red, house}})}$

Note that the context of the first n – 1 n-grams is filled with start-of-sentence markers, typically denoted <s>.

Additionally, without an end-of-sentence marker, the probability of an ungrammatical sequence *I saw the would always be higher than that of the longer sentence I saw the red house.

Maximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is

${\displaystyle P(w_{m}|w_{1},\ldots ,w_{m-1})={\frac {1}{Z(w_{1},\ldots ,w_{m-1})}}\exp(a^{T}f(w_{1},\ldots ,w_{m}))}$

where ${\displaystyle Z(w_{1},\ldots ,w_{m-1})}$ is the partition function, ${\displaystyle a}$ is the parameter vector, and ${\displaystyle f(w_{1},\ldots ,w_{m})}$ is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on ${\displaystyle a}$ or some form of regularization.

The log-bilinear model is another example of an exponential language model.

Neural language models (or continuous space language models) use continuous representations or embeddings of words to make their predictions.[8] These models make use of Neural networks.

Continuous space embeddings help to alleviate the curse of dimensionality in language modeling: as language models are trained on larger and larger texts, the number of unique words (the vocabulary) increases.[lower-alpha 1] The number of possible sequences of words increases exponentially with the size of the vocabulary, causing a data sparsity problem because of the exponentially many sequences. Thus, statistics are needed to properly estimate probabilities. Neural networks avoid this problem by representing words in a distributed way, as non-linear combinations of weights in a neural net.[9] An alternate description is that a neural net approximates the language function. The neural net architecture might be feed-forward or recurrent, and while the former is simpler the latter is more common.

Typically, neural net language models are constructed and trained as probabilistic classifiers that learn to predict a probability distribution

${\displaystyle P(w_{t}|\mathrm {context} )\,\forall t\in V}$.

I.e., the network is trained to predict a probability distribution over the vocabulary, given some linguistic context. This is done using standard neural net training algorithms such as stochastic gradient descent with backpropagation.[9] The context might be a fixed-size window of previous words, so that the network predicts

${\displaystyle P(w_{t}|w_{t-k},\dots ,w_{t-1})}$

from a feature vector representing the previous k words.[9] Another option is to use "future" words as well as "past" words as features, so that the estimated probability is

${\displaystyle P(w_{t}|w_{t-k},\dots ,w_{t-1},w_{t+1},\dots ,w_{t+k})}$.

This is called a bag-of-words model. When the feature vectors for the words in the context are combined by a continuous operation, this model is referred to as the continuous bag-of-words architecture (CBOW).[10]

A third option that trains slower than the CBOW but performs slightly better is to invert the previous problem and make a neural network learn the context, given a word[10]. More formally, given a sequence of training words ${\displaystyle w_{1},w_{2},w_{3},\dots ,w_{T}}$, one maximizes the average log-probability

${\displaystyle {\frac {1}{T}}\sum _{t=1}^{T}\sum _{-k\leq j\leq k,j\neq 0}\log P(w_{t+j}|w_{t})}$

where k, the size of the training context, can be a function of the center word ${\displaystyle w_{t}}$. This is called a skip-gram language model.[11] Bag-of-words and skip-gram models are the basis of the word2vec program.[12]

Instead of using neural net language models to produce actual probabilities, it is common to instead use the distributed representation encoded in the networks' "hidden" layers as representations of words; each word is then mapped onto an n-dimensional real vector called the word embedding, where n is the size of the layer just before the output layer. The representations in skip-gram models have the distinct characteristic that they model semantic relations between words as linear combinations, capturing a form of compositionality. For example, in some such models, if v is the function that maps a word w to its n-d vector representation, then

${\displaystyle v(\mathrm {king} )-v(\mathrm {male} )+v(\mathrm {female} )\approx v(\mathrm {queen} )}$

where ≈ is made precise by stipulating that its right-hand side must be the nearest neighbor of the value of the left-hand side.[10][11]

A positional language model[13] assesses the probability of given words occurring close to one another in a text, not necessarily immediately adjacent. Similarly, bag-of-concepts models[14] leverage the semantics associated with multi-word expressions such as buy_christmas_present, even when they are used in information-rich sentences like "today I bought a lot of very nice Christmas presents".

Despite the limited successes in using neural networks[15], authors acknowledge the need for other techniques when modelling sign languages.

• Bidirectional Encoder Representations from Transformers
• CSLM – Free toolkit for feedforward neural language models
• DALM – Fast, Free software for language model queries
• Generative Pre-trained Transformer
• IRSTLM – Free software for language modeling
• Kylm (Kyoto Language Modeling Toolkit) – Free language modeling toolkit in Java
• KenLM – Fast, Free software for language modeling
• LMSharp – Free language model toolkit for Kneser–Ney-smoothed n-gram models and recurrent neural network models
• MITLM – MIT Language Modeling toolkit. Free software
• NPLM – Free toolkit for feedforward neural language models
• OpenGrm NGram library – Free software for language modeling. Built on OpenFst.
• OxLM – Free toolkit for feedforward neural language models
• Positional Language Model
• RandLM – Free software for randomised language modeling
• RNNLM – Free recurrent neural network language model toolkit
• SRILM – Proprietary software for language modeling
• VariKN – Free software for creating, growing and pruning Kneser-Ney smoothed n-gram models.
• Language models trained on newswire data
• Web Page NGram Viewer