Kneser-Ney smoothing with a correcting function for small data sets

Опубликовано 6 сентября 2016, 16:55

We present a technique which improves the Kneser- Ney smoothing algorithm on small data sets for bigrams and we develop a numerical algorithm which computes the parameters for the heuristic formula with a correction. We give motivation for the formula with correction on a simple example. Using the same example we show the possible difficulties one may run into with the numerical algorithm. Applying the algorithm to test data we show how the new formula improves the results on cross-entropy.