CORSIS

As part of a research project I am involved in, the development version of CORSIS is now being used and actively extended with latest technologies. For more information, visit our new wiki .

Here is a cleaned-up performance test for several different implementations of levenshtein I have blogged about recently. This test was emailed to me by Ahmed Ghoneim, who has also kindly agreed to its publication on my blog. I am very grateful to him for his excellent contribution. I have slightly altered his file to do away with the unnecessary local variables in my C2C# port of the GNULevenshtein method. I would like to hear from you which methods perform best on your machine. Please drop a comment ^_^! LevenshteinAlgorithmPerformanceTest.cs code only Packages code, data and sample binary in zip and self-executable zip formats Please note that the GNULevenshtein method was found to be buggy! Here is the new replacement method .

I've come across another limitation of WordSmith Tools 4. It can't process large files. And what do you think is 'large' for WST4? Well... it can be a file as small as 9,35 MB depending on the type of data stored. While reading Wikipedia, I stumbled upon the GIMPS Project at http://www.mersenne.org/ and found their 44th Mersenne Prime discovery: http://www.mersenne.org/prime10.txt - a 9,808,358 digits long integer number. I wanted to see how the Wordlister of Tenka Text 0.1.3 performed with such a large single token and in fact I expected it to throw some sort of unexpected exception... to be continued

Tenka Text 0.1.3* on Windows Vista Ultimate (.NET 2.0>) New Update Dialog from Tenka Text 0.1.3* on Windows Vista Ultimate Tenka Text 0.1.3* on Debian 4.0 (Mono 1.2.4 preview 3) SVN267 0.1.3 Release Notes This version of Tenka Text introduces dynamism into Wordlister. It’s the first release that juggles with interface design ideas which have come to prove themselves in professional integrated development environments. The main feature of this release is the explorer control in Wordlister, which is a tree view that performs organizational tasks much like the solution explorer control in Visual Studio. Wordlister explorer has been implemented in such a way that enables you to add, remove or switch between frequency lists dynamically. Because graphical user interfaces of integrated development environments are designed for highly complicated usage scenarios and heavy workloads developers tend to have, they stand as great examples for computational linguists that aim to deliver equally ...

Here is a screenshot from a new feature which is going to be available in the wordlister tool of the next release: study explorer - a tree view of the files you are analyzing which helps you dynamically change between result views back and forth. Close to the alpha or beta, probably all the functionality will have been combined in just one type of window. Things may end up looking like how Visual Studio does since 2003. I hope a fully-portable docking suite becomes available until then to make things easy.

Tenka Text 0.1.1 on Windows Vista Ultimate This is the first versioned release of Tenka Text. CLASS LIBRARY The class library has undergone great changes one of which are the brand new segmenter customization classes which can be seen at: http://corsis.svn.sourceforge.net/viewvc/corsis/trunk/Tenka.Text/Tenka.Text/Segmentation/Builder.cs?revision=197&view=markup&pathrev=197 and read about at: http://tenkatext.blogspot.com GRAPHICAL USER INTERFACE Main Window 1) Improved select files dialog. 2) New options dialog. Wordlister 1) Segmenter Customization: Segmenters used to create frequency lists are now dynamically compiled. (Compilation uses reflection emit and MSIL/CIL/IL assembly to ensure maximum performance.) 2) Export Feature: You can now export/save data from the wordlister in either plain text or xml format. (This feature is also supported on Mono 1.2.3.50.) 3) Wordlister is now at alpha stage: With the addition of the new segmenter compiler and a simple export feature, Te...

I've been trying to integrate reflection emit into Tenka Text recently and that's how far I have come in code: Builder.cs . You can use reflection emit to compile and build types at runtime. A pretty amazing way of using this technology is having an abstract class whose implementation you provide at runtime. Tenka Text is going to use reflection emit to provide custom segmentation. Using reflection emit instead of going for a simpler approach has numerous distinguishing advantages. I quote my friend Mike Scott from the documentation of his WordSmith Tools 4 here: [...] you may wish to allow certain additional characters within a word. For example, in English, the apostrophe in father's is best included as a valid character as it will allow processing to deal with the whole word instead of cutting it off short. (If you change language to French you might not want apostrophes to be counted as acceptable mid-word characters.) Examples: ' (only apostrophes allowed in the mi...

And now I am writing from the brand new Windows Vista Ultimate. I have to say that I truly like this OS and because I bought the OEM Version, it only cost me 189€. I will be playing around with it for the next couple of days before going back to the development of Tenka Text. Oh by the way... one thing I am truly delighted about is that it does not show any user access control warnings if you try to run Tenka Text*. Managed applications are priviliged on Vista.

I spent the night implementing the option I mentioned in the previous post and decided to release a new binary after 22 days. (Corresponding SVN revision: 96) Tenka Text WordLister is now more precise than WordSmith Tools WordList. It has an option now through which a user can choose to display in the statistics view for frequency lists either the count of tokens that are a certain number of characters long or the count of types that are a certain number of characters long. *NEW* WordSmith Tools 4 can only display the first but not the latter. Check all three screenshots! Tenka Text WordLister with Length Statistics for Tokens Tenka Text WordLister with Length Statistics for Types WordSmith Tools 4 with Length Statistics for Words(Tokens)

^_^ Here is the new statistics view for frequency lists from the 2007-02-05 pre-alpha release. I'm still actively working on frequency lists at GUI and lower levels and some of the following features might be ready for the next release: An option to have the program display in the statistics view for frequency lists either the count of tokens that are a certain number of characters long or the count of types that are a certain number of characters long. * Tenka Text 2007-02-05 and WordSmith Tools 4 can only display the first but not the latter. There is only a little GUI work to deliver this option. Live updates. Once in place, users will be able to add or remove files from their frequency lists without the need to recalculate everything from the start. Well, that's all the news about Tenka Text 2007-02-05. Let's move on to more personal stuff. Logographic Failure I found out that the "Tenka" Logo 「天花」, which was my personal attempt at creating a digital seal with...

TT5 -> TT8 performance difference I conducted a performance test on one of the remotely accesible computers of the University of Heidelberg. (2 physical/4 logical cpus and 2 GB Ram) The test was performed by creating a frequency list based on the helsinki corpus (9.793 KB, single text file) and then sorting it. As you can see below, my optimization efforts seem to have paid off well. WordSmith Tools 4.0.0.374 took about 13 seconds to create and sort a word list into: Alphabetical order Frequency order Alphabetical order between types with the same frequency value TT5 (svn revision 17, binary release: 2006-11-25) required 2,51 seconds to create and 10,07 seconds to sort the list into: Alphabetical order Frequency order TT8 (svn revision 66) needed 1,10~ seconds to create and 2,40~ seconds to sort the list into: Alphabetical order Frequency order Alphabetical order between types with the same frequency value * Performance Comparison Table WS4 0.0.374 TT5 SVN17 TT8 SVN62 TT8.1 S...

I have renamed parts of the class library. Tenka.Text.Segmentation This is the new name of the namespace which was previously called Tenka.Text.Enumerators. I will be providing some default segment enumerator implementations until I have bought and read a good book on IL, which will mark the turn of events paving the way to runtime-compiled customizable segmentation. I found a way to do this using C# syntax at http://www.codeproject.com/cs/algorithms/matheval.asp . The new frequency list is also almost done and will completely replace the old one. I have a hard time doing the necessary encapsulation work because I keep thinking that I might be doing something that might adversely affect the performance.

To do a quick test on performance improvement, I redid some parts of my counter and switched to the new high performance hashset collection from the Orcas January CTP. I used the helsinki corpus (9.793 KB, single text file) and conducted a single-word frequency list operation (create and sort). And here are the results: TT5 SVN17 TT7 SVN47 Performance Improvement Create 2,17 1,84 1,17x Sort 12,42 2,28 5,44x Display 0,80~ 0,80~ - Total 15,39~ 4,92~ 3,12x For your information, WordSmith Tools 4 (4.0.0.374) takes about 23 seconds to perform the same operation.

I tested performances of several set implementations. A set is an unordered collection of unique elements. Time required to create a set of unique words with different implementations (Corpus Size: 278,675 tokens of 21,828 types): a cheap set implementation which derives from the BCL generic list collection and imposes Contains(T item) checks on each add / insert operation [ SVN ] 21,20~ seconds a set implementation which is a reflected / disassembled partial copy of the BCL generic list collection and performs manually inlined Contains checks on each add / insert operation 20,70~ seconds System.Collections.Specialized.StringCollection : if (!set.Contains(word)) set.Add(word); 18.96~ seconds HashSet from Orcas October CTP 0,17~ seconds ^_^ just who can beat this? I immediately decided to switch to the new generic HashSet from the BCL guys. Tenka.Text will greatly benefit from this development especially when sorting word-frequency dictionaries* on their frequencies. (* Your ...

I finished reading Framework Design Guidelines . It was one insightful book well worth reading. Through it, I came to learn that even in the .NET class libraries are some unsolved/unnoticed design issues which shipped with the product - naming of some classes for example or the fact that StringBuilder remains to reside in the namespace System.Text and not in System. I am happy to have read this book at the very early stages of my program's development. I have the freedom to completely redo the design of my class libraries now. In fact I have already downloaded FxCop to analyse the codes I produced so far. The list of possible issues was longer than the longest of poems. I will be looking into the matter for the next few weeks. I will also go see how regular expressions use System.Reflection.Emit to compile regex instances on-the-fly.

Hi, this is my first blog post. I have established this blog to keep people in the know about my personal open-source corpus analysis project Tenka Text . It is my, Cetin Sert's, open-source to Mike Scott's WordSmith Tools (4). As you can click the above link to read and get an introductory idea about what or what not my project is, I will allow myself to skip an introduction and tell only the most recent development news. Back to Basics Well... first of all, I have decided to refocus on the sub-gui parts of the code once again. I saw a need to rethink some of the basic design elements like the overall inheritance/relation model of the classes and structs in the Tenka.Text namespace. Before going back on to the GUI development, I want to simplify the word enumerators, do away with the bulky interfaces, learn more about the available inlining optimizations, move from public fields to public properties and minimize the public exposure of pointers. System.Reflection.Emit & Run...

Contact Cetin Sert Heidelberg Baden Württemberg, Germany cetin.sert@gmail.com [email] nomadsoul@msn.com [messenger] Personal Pages chetin.deviantart.com My free 3D models on Turbosquid