Speed up a slow loop

[Alex Tweedly]

On 08/03/2022 07:27, Neville Smythe via use-livecode wrote:
> I believe there were two problems: a) the original search search algorithm was too slow, and b) when implemented on certain devices the app went into an unusable state. Both problems were caused by the wordlist being stored being too large (275K words for a file size of  2.5 MB). The second issue we are pretty sure was Virtual Memory swap space thrashing.

I guess I'm not convinced about VM space thrashing being the problem, 
certainly not due to the wordlist. It's 2.5 Mb - i.e. 0.02% of the real 
RAM in a Pixel 5. There may be something else in the app making the 
total VM space very large - but the wordlist wouldn't do it on its own.


> An alternative is to use an sqlite database, always an option which should be considered when handling large data. This completely removes the issue of having the wordlist in memory, and at the same time provides an extremely fast search engine, so was worth exploring as a solution to both problems.  Storing the words in a db single table with an index on initial letter and word length, or as lots of tables, one for each initial and length (!), both return a result for a search for a word in a small fraction of a millisecond, so definitely this would be a viable solution which would handle all letter distributions.
Sounds cool. I'd like to see an example of how this would be created and 
used (I'm very much a SQL novice).
> There is however a downside: both methods produce a db file size of 7.1 MB for the SOWPODS wordlist, which rather bloats the app footprint, even when you discount the text file version of the wordlist which no longer needs to be stored.
You could do the opposite. Store the wordlist (700Kb compressed), and 
decompress/populate into the SQLite database on initial run.
> But now, speaking of databases, I have a question. I have an update to my nsScriptDatabase stack which I want to upload to the Sample Stacks. Because it really should be compiled to a standalone, I really need to upload a pair of stacks, a launch stack and one to hold data which can one modified by the user. But it would seem a sample stack must be a single item. What to do?

Yep -this is a big drawback of the Sample Stacks system.

Maybe you could make a single app, which contains the two desired stacks 
as custom props.

(Sometimes it's easier to type code than to describe it :-)
In your preparation, you'd do:
  put URL ("binfile:launchstack.livecode") into tmp
  set the cLaunch of me to compress(tmp)
  put URL ("binfile:userdatastack.livecode") into tmp
  set the cUser of me to compress(tmp)

And then when the user downloads your sample stack, it does the inverse 
to create the two local stacks, with an info box telling them about it.

Alex.