CORSIS

Mike Scott has finally released a public beta of WordSmith Tools 5.0 . I want to quote him on a few points: System Requirements WordSmith Tools 5 is for Windows 2000 or later. It will be happiest on a fairly modern PC (e.g bought in the last 4 years) with plenty of memory and hard disk space. Or an Intel Mac running Windows . Now that's an interesting world view we have here. What's an Intel Mac anyway? It's a PC produced by Apple. So this sentence actually reads: System Requirements: a PC running Windows or a PC (produced by Apple) running Windows. Technically deep Mr. Scott seems to be. -__-" Here is my favourite one: What's new in version 5 WordSmith is organic software ! Version 4 was a complete new re-write. Since it was launched in 2004, numerous re-compilations of WS4 were issued (about one a week on average), sometimes with a very small bug-fix but other times incorporating changes users had suggested. At the same time the Help was updated. Version 5.0 was...

I've come across another limitation of WordSmith Tools 4. It can't process large files. And what do you think is 'large' for WST4? Well... it can be a file as small as 9,35 MB depending on the type of data stored. While reading Wikipedia, I stumbled upon the GIMPS Project at http://www.mersenne.org/ and found their 44th Mersenne Prime discovery: http://www.mersenne.org/prime10.txt - a 9,808,358 digits long integer number. I wanted to see how the Wordlister of Tenka Text 0.1.3 performed with such a large single token and in fact I expected it to throw some sort of unexpected exception... to be continued

Here is a screenshot from a new feature which is going to be available in the wordlister tool of the next release: study explorer - a tree view of the files you are analyzing which helps you dynamically change between result views back and forth. Close to the alpha or beta, probably all the functionality will have been combined in just one type of window. Things may end up looking like how Visual Studio does since 2003. I hope a fully-portable docking suite becomes available until then to make things easy.

Tenka Text 0.1.1 on Windows Vista Ultimate This is the first versioned release of Tenka Text. CLASS LIBRARY The class library has undergone great changes one of which are the brand new segmenter customization classes which can be seen at: http://corsis.svn.sourceforge.net/viewvc/corsis/trunk/Tenka.Text/Tenka.Text/Segmentation/Builder.cs?revision=197&view=markup&pathrev=197 and read about at: http://tenkatext.blogspot.com GRAPHICAL USER INTERFACE Main Window 1) Improved select files dialog. 2) New options dialog. Wordlister 1) Segmenter Customization: Segmenters used to create frequency lists are now dynamically compiled. (Compilation uses reflection emit and MSIL/CIL/IL assembly to ensure maximum performance.) 2) Export Feature: You can now export/save data from the wordlister in either plain text or xml format. (This feature is also supported on Mono 1.2.3.50.) 3) Wordlister is now at alpha stage: With the addition of the new segmenter compiler and a simple export feature, Te...

I've been trying to integrate reflection emit into Tenka Text recently and that's how far I have come in code: Builder.cs . You can use reflection emit to compile and build types at runtime. A pretty amazing way of using this technology is having an abstract class whose implementation you provide at runtime. Tenka Text is going to use reflection emit to provide custom segmentation. Using reflection emit instead of going for a simpler approach has numerous distinguishing advantages. I quote my friend Mike Scott from the documentation of his WordSmith Tools 4 here: [...] you may wish to allow certain additional characters within a word. For example, in English, the apostrophe in father's is best included as a valid character as it will allow processing to deal with the whole word instead of cutting it off short. (If you change language to French you might not want apostrophes to be counted as acceptable mid-word characters.) Examples: ' (only apostrophes allowed in the mi...

I spent the night implementing the option I mentioned in the previous post and decided to release a new binary after 22 days. (Corresponding SVN revision: 96) Tenka Text WordLister is now more precise than WordSmith Tools WordList. It has an option now through which a user can choose to display in the statistics view for frequency lists either the count of tokens that are a certain number of characters long or the count of types that are a certain number of characters long. *NEW* WordSmith Tools 4 can only display the first but not the latter. Check all three screenshots! Tenka Text WordLister with Length Statistics for Tokens Tenka Text WordLister with Length Statistics for Types WordSmith Tools 4 with Length Statistics for Words(Tokens)

I tested performances of several set implementations. A set is an unordered collection of unique elements. Time required to create a set of unique words with different implementations (Corpus Size: 278,675 tokens of 21,828 types): a cheap set implementation which derives from the BCL generic list collection and imposes Contains(T item) checks on each add / insert operation [ SVN ] 21,20~ seconds a set implementation which is a reflected / disassembled partial copy of the BCL generic list collection and performs manually inlined Contains checks on each add / insert operation 20,70~ seconds System.Collections.Specialized.StringCollection : if (!set.Contains(word)) set.Add(word); 18.96~ seconds HashSet from Orcas October CTP 0,17~ seconds ^_^ just who can beat this? I immediately decided to switch to the new generic HashSet from the BCL guys. Tenka.Text will greatly benefit from this development especially when sorting word-frequency dictionaries* on their frequencies. (* Your ...

Hi, this is my first blog post. I have established this blog to keep people in the know about my personal open-source corpus analysis project Tenka Text . It is my, Cetin Sert's, open-source to Mike Scott's WordSmith Tools (4). As you can click the above link to read and get an introductory idea about what or what not my project is, I will allow myself to skip an introduction and tell only the most recent development news. Back to Basics Well... first of all, I have decided to refocus on the sub-gui parts of the code once again. I saw a need to rethink some of the basic design elements like the overall inheritance/relation model of the classes and structs in the Tenka.Text namespace. Before going back on to the GUI development, I want to simplify the word enumerators, do away with the bulky interfaces, learn more about the available inlining optimizations, move from public fields to public properties and minimize the public exposure of pointers. System.Reflection.Emit & Run...