Speed up a slow loop

[Neville Smythe]

Jacque seems to have found a satisfactory solution to her problem with the help of the excellent suggestions we have seen here. Since there seems to have been a lot of interest, I thought I would take the opportunity to report on an alternative method I had been exploring with Jacque via email.

I believe there were two problems: a) the original search search algorithm was too slow, and b) when implemented on certain devices the app went into an unusable state. Both problems were caused by the wordlist being stored being too large (275K words for a file size of  2.5 MB). The second issue we are pretty sure was Virtual Memory swap space thrashing. 

Using Quentin’s why-didn’t-I-think-of-that filtering suggestion the wordlist can be reduced (in the vasty majority of cases) to a manageable size, say 20K words or less, and both problems solved very elegantly.

There are however edge cases where the filtered wordlist can still be huge, over 200K words, [ eg unused letters are JQXYZ  ]. Such cases are going to be very rare, so Jacque is going with the above solution. 

An alternative is to use an sqlite database, always an option which should be considered when handling large data. This completely removes the issue of having the wordlist in memory, and at the same time provides an extremely fast search engine, so was worth exploring as a solution to both problems.  Storing the words in a db single table with an index on initial letter and word length, or as lots of tables, one for each initial and length (!), both return a result for a search for a word in a small fraction of a millisecond, so definitely this would be a viable solution which would handle all letter distributions.

There is however a downside: both methods produce a db file size of 7.1 MB for the SOWPODS wordlist, which rather bloats the app footprint, even when you discount the text file version of the wordlist which no longer needs to be stored. If one was writing a crossword app, where you might be searching for words for which you know the 3rd and fifth letter for example, bread and butter for an sqlite query, this would definitely be the way to go, but probably overkill for Jacque’s app.

But now, speaking of databases, I have a question. I have an update to my nsScriptDatabase stack which I want to upload to the Sample Stacks. Because it really should be compiled to a standalone, I really need to upload a pair of stacks, a launch stack and one to hold data which can one modified by the user. But it would seem a sample stack must be a single item. What to do?

Neville