CORSIS

sample plain text in Devanāgarī script character frequency analysis results source code sample program binaries *You can use the sample program not only with devanagari but with other scripts as well. See example. A student of final year B.Tech at National Institute of Technology, Hamirpur, India wrote to me today: (emphasis added) I came across Tenka-text while searching through the net for existing concordance softwares. I wish to develop a similar kind of free software for the Dev Nagri script which is the script for Hindi, our national Language. Initially I plan to develop just the character frequency calculation functionality which is not provided by any currently available product . Could you please guide me a little on how to go about this. I shall value your suggestions very much. To which I first replied: ... I'm at work right now and thus do not have access to my development tools and personal library and because of this cannot reply with an example C# program immediately...

I've come across another limitation of WordSmith Tools 4. It can't process large files. And what do you think is 'large' for WST4? Well... it can be a file as small as 9,35 MB depending on the type of data stored. While reading Wikipedia, I stumbled upon the GIMPS Project at http://www.mersenne.org/ and found their 44th Mersenne Prime discovery: http://www.mersenne.org/prime10.txt - a 9,808,358 digits long integer number. I wanted to see how the Wordlister of Tenka Text 0.1.3 performed with such a large single token and in fact I expected it to throw some sort of unexpected exception... to be continued

^_^ Here is the new statistics view for frequency lists from the 2007-02-05 pre-alpha release. I'm still actively working on frequency lists at GUI and lower levels and some of the following features might be ready for the next release: An option to have the program display in the statistics view for frequency lists either the count of tokens that are a certain number of characters long or the count of types that are a certain number of characters long. * Tenka Text 2007-02-05 and WordSmith Tools 4 can only display the first but not the latter. There is only a little GUI work to deliver this option. Live updates. Once in place, users will be able to add or remove files from their frequency lists without the need to recalculate everything from the start. Well, that's all the news about Tenka Text 2007-02-05. Let's move on to more personal stuff. Logographic Failure I found out that the "Tenka" Logo 「天花」, which was my personal attempt at creating a digital seal with...

TT5 -> TT8 performance difference I conducted a performance test on one of the remotely accesible computers of the University of Heidelberg. (2 physical/4 logical cpus and 2 GB Ram) The test was performed by creating a frequency list based on the helsinki corpus (9.793 KB, single text file) and then sorting it. As you can see below, my optimization efforts seem to have paid off well. WordSmith Tools 4.0.0.374 took about 13 seconds to create and sort a word list into: Alphabetical order Frequency order Alphabetical order between types with the same frequency value TT5 (svn revision 17, binary release: 2006-11-25) required 2,51 seconds to create and 10,07 seconds to sort the list into: Alphabetical order Frequency order TT8 (svn revision 66) needed 1,10~ seconds to create and 2,40~ seconds to sort the list into: Alphabetical order Frequency order Alphabetical order between types with the same frequency value * Performance Comparison Table WS4 0.0.374 TT5 SVN17 TT8 SVN62 TT8.1 S...

While investing some time in improving my new frequency list implementation, I also started to watch some new Japanese Dramas from Winter 2007 : Erai Tokoro ni Totsuide Shimatta! Hana Yori Dango 2 Enka* no Joou I just fell in love with the songs in Enka no Joou and decided to cut and retime subs of one the enkas in it. Here you can download this wonderful song. Two great open-source programs helped me do the work in under 2 minutes: VirtualDub and Aegisub . Both are extremely easy to use!

To do a quick test on performance improvement, I redid some parts of my counter and switched to the new high performance hashset collection from the Orcas January CTP. I used the helsinki corpus (9.793 KB, single text file) and conducted a single-word frequency list operation (create and sort). And here are the results: TT5 SVN17 TT7 SVN47 Performance Improvement Create 2,17 1,84 1,17x Sort 12,42 2,28 5,44x Display 0,80~ 0,80~ - Total 15,39~ 4,92~ 3,12x For your information, WordSmith Tools 4 (4.0.0.374) takes about 23 seconds to perform the same operation.