CORSIS

While implementing the perplexity function on Markov n-gram models as it is described on page 14 of Jurafsky & Martin's SLP to appear , I came across some floating point overflow, underflow issues and had to come up with equations to avoid them. Here is my solution in detail and its bigram implementation . It took me a lot of whiteboarding and a few hours to figure this one out but the resulting libcorsis code is %100 foreign intellectual property free ^_^. I am now looking for ways to analyze distributions graphically and testing different encapsulations of probability values and their interaction with the public API.

Here is a very short update on what I have been doing recently. 1) Thanks to the excellent quality of the introduction to computational linguistics course I am taking this term , I get to learn a great many new stuff which boost the development tempo of my library. Here is a particularly interesting case: Markov N-Gram Models . These are soon going to find some interesting application in my experimental HMM-based PoS-Tagger. 2) C# 3.0 and .NET 3.5 have been released. I'm truly astonished with the improved ease of programming this iteration delivers. Waiting eagerly to lay my hands on some quality books on both C# and .NET 3.5. 3) F#! After watching Dr. Brian Beckman's latest video on Channel 9 about monads I decided to learn F#. Parallelization libraries like Parallel FX might provide all that is needed to seamlessly parallelize execution but learning something natively functional seems to be a great asset for the very near future. 4) I had the chance yesterday to attend a ...

While reading some interesting stuff about minimum edit distances in preparation for today's lecture ( ECL/ICL ), which is just about 45 minutes ahead in time as I'm writing this, I had the chance to test 5 different implementations of the Levenshtein minimum edit distance algorithm. Here is a screenshot first: I'll get into details later but let me announce the winner! And the winner is ... gLDp! gLDp is a funny display name for a levenshtein implementation from a C project. original implementation in C: levenshtein.c my C# port: libcorsis code C vs CIL vs C# Now I want to get mercilessly picky with my own port and today's C# compilers. The ternary conditional expressions in my port (lines: #516 , #524 , #533 ) are there to circumvent the following restriction: // valid C int x = 0; int y = 0; int z = 0; z += x == y; x, y and z are initialized 0 and in the final line z gets incremented by 1. This is valid code in C but causes a compile-time error in C#: C# does no...

Tenka Text has been rebranded as CORSIS. Following addresses are no longer accessible: http://tenkatext.sourceforge.net http://www.sourceforge.net/projects/tenkatext You can reach the project on sourceforge.net by searching for "CORSIS" . The new domain name of the project is www.corsis.de. First new binaries are expected to be available in 2008 Q1.

A quick update on API refactorings! Here is a snapshot of what the code examined in the last post would look like with refactorings and improvements I have made so far: Before: the code examined in the last post After: For segmenting streams into, say, words for example, one could also use something like SED on GNU/Linux, some regular expressions implementation of a programming language or whatever. So why am I such an otaku ? Why not just go with the given the naive and easy way? Well, I am a performance and control freak and CIL is great fun and I feel 'pleasure' writing assembly code for a VM but most importantly, Benihime, 紅姫 makes the perfect training ground for learning language and compiler design. Prior to getting into deep hack mode on Benihime, 紅姫, I had no idea about the differences between 'expressions', 'statements', 'branches' or 'stacks'. Implementing complex boolean expressions in conditional statements like if((c && pm) ...

An interesting design concern has brought the development of my new segmenter compiler to a temporary standstill for tonight: parallelization . I was trying to refactor and improve the design of my new segmenter compiler Benihime, 紅姫. (It is named after Urahara's sword from the Japanese anime Bleach . Benihime means crimson princess , what is more suitable to call a state-of-the-art segmenter. ^o^ theheee~~) One of my major concerns with the new implementation was decoupling the flow control logic from the segmenter builder and the flow direction. This is essential for being able to reuse the same logic to compile two segmenters that run in opposite directions for example. Let me illustrate the problem with the help of a file I happened to submit as a practice for our introduction to computational linguistics course just 5 days ago: // /home/sert/Projects/hw1/hw1/Main.cs created with MonoDevelop // // project created on 10/21/2007 at 3:19 AM using System; using System.IO; using...

Corsis is seeking code reviewers, documentation writers and framework designers. If you want to work with us on non-commercial, voluntary terms, apply for a position today! Project Openings Corsis started as an open-source answer to commercial corpus analysis software and now constitutes a test bed for research in various subjects such as compiler design , performance optimization and parallel computation .