Speed up a slow loop

[Neville Smythe]

Bob’s suggestion of storing the word list in an sqlite database would keep the list on disk and not in memory, so avoiding VM swapping which is  I think  the basic problem, and also use a much faster search engine than LC to do the checking for the existence of user candidates. A more elegant solution than breaking up the wordlist into hundreds of small files. 

It is possible because we know the structure of the data in this case that lots of small files, with a couple of other tricks to reduce file size, and then using an LC search, could be faster (because the OS file system is even more efficient than sqlite) but I’d be surprised if the db approach didn’t solve your problem.


Neville