Perplexity in Markov N-Gram Models

While implementing the perplexity function on Markov n-gram models as it is described on page 14 of PDFJurafsky & Martin's SLP to appear, I came across some floating point overflow, underflow issues and had to come up with equations to avoid them.

Here is PDFmy solution in detail and C#its bigram implementation.

It took me a lot of whiteboarding and a few hours to figure this one out but the resulting libcorsis code is %100 foreign intellectual property free ^_^.

I am now looking for ways to analyze distributions graphically and testing different encapsulations of probability values and their interaction with the public API.

Comments

Post a Comment

Popular posts from this blog

Levenshtein Distance Algorithm: Fastest Implementation in C#

Image
Here is a cleaned-up performance test for several different implementations of levenshtein I have blogged about recently. This test was emailed to me by Ahmed Ghoneim, who has also kindly agreed to its publication on my blog. I am very grateful to him for his excellent contribution. I have slightly altered his file to do away with the unnecessary local variables in my C2C# port of the GNULevenshtein method. I would like to hear from you which methods perform best on your machine. Please drop a comment ^_^! LevenshteinAlgorithmPerformanceTest.cs code only Packages code, data and sample binary in zip and self-executable zip formats Please note that the GNULevenshtein method was found to be buggy! Here is the new replacement method .
Read more

WordSmith Tools 5.0, Tenka Text in China

Mike Scott has finally released a public beta of WordSmith Tools 5.0 . I want to quote him on a few points: System Requirements WordSmith Tools 5 is for Windows 2000 or later. It will be happiest on a fairly modern PC (e.g bought in the last 4 years) with plenty of memory and hard disk space. Or an Intel Mac running Windows . Now that's an interesting world view we have here. What's an Intel Mac anyway? It's a PC produced by Apple. So this sentence actually reads: System Requirements: a PC running Windows or a PC (produced by Apple) running Windows. Technically deep Mr. Scott seems to be. -__-" Here is my favourite one: What's new in version 5 WordSmith is organic software ! Version 4 was a complete new re-write. Since it was launched in 2004, numerous re-compilations of WS4 were issued (about one a week on average), sometimes with a very small bug-fix but other times incorporating changes users had suggested. At the same time the Help was updated. Version 5.0 was
Read more

Mono 1.2.5 binaries for Solaris 10/x86

I think I've finally managed to botch it all up. I created a JRE-style zipped package, which you can download here 1,2 at your own risk ;). Whoa, it's taken me 4 days just to get so far with mono/solaris. ^o^ I wanna sleep like the relakkuma now hehe... These binaries are compiled for 32-bit/x86 processors. They also run on 64-bit/x86-64 processors. They are not compiled for running winforms applications. To extract the archive use: "gtar -zxvf"
Read more