large arrays

Fri Sep 19 22:02:23 EDT 2014

Hello Richard,
I had to leave for awhile but now here is the updated info:
Using put len(arrayEncode(myArray)) I got the following:
there are actually 11 different arrays - one for each of the word lengths 
from 2 to 12 letters.
The 3-ltr word array is 17672
The 9-ltr word array is 1159610 (there are a lot more 9-ltr words than 3-ltr 
words.)
I didn't do all the others, but that will give you an idea.  7 and 8-ltr 
words will be close to 9-ltr words in size

I haven't yet had time to test with later versions, but once I do, I will 
let you know.
The problem is not the time it takes to load the arrays.  As I mentioned, it 
loads all 11 in just over 1 second.
Once they are loaded, I access the data directly from the arrays and not 
from the fields.
The number elements (lines) in the arrays varies.  The 3-ltr array has 1292 
lines - I guess that means elements.
The 9-ltr array has 40,727 lines or elements.
Each line is in a format like this:
apt,APT,PAT,TAP  where "apt" is the key, which only needs to be found once 
and it shows all the 3-ltr words that can be made with those letters.  Of 
course, that speeds up the search process tremendously.

The time (about .4 seconds) I mentioned for finding the 596 words found in 
ELEPHANTS is what I tested today.  But I know that even on a slower computer 
using the stack built with Rev 2.9, the time was very close to that - about 
1/2 of a second.

About 5 or 6 years ago when I was using 2.9 and then upgraded to 3.5, I not 
only spoke to someone at RunRev (forgot who) but I also spoke to Jerry 
Daniels, of Daniel and Mara, on the phone about several RunRev things and we 
discussed the array issue.  He acknowledged that with the release of 3.5 
there were some serious array issues, that RunRev knew about them and Jerry 
hoped that they would soon be fixed.  They were not fixed with the release 
of RunRev 4.0 and the only version of LC that I've used extensively is 6.1.1 
and the array issues are still not fixed in that release - although I would 
say it is a big improvement over RunRev 3.5.

The crazy thing is this:  I haven't made any significant changes to my stack 
over the years - mostly cosmetic stuff and the array sizes are what they've 
always been.  RunRev 2.9 never flinched, brought up the data in the arrays 
almost instantly.  But since then it has been worse than trying to get my 
little grandson to wear a shirt.

I'm working on other projects now, but wanted to incorporate some of the 
anagram features in my latest project.  I am very reluctant to do so because 
the complete sluggishness of it makes it way too frustrating to try and 
program.

Someday soon I will get around to trying LC 7.0 or 7.1 or whatever is there 
and HOPEFULLY the issue will be solved.

In the meantime, thank you and Jacqueline SO MUCH for your kind attention 
and offers of help.

All the best,
Larry

----- Original Message ----- 
From: "Richard Gaskin" <ambassador at fourthworld.com>
To: <use-livecode at lists.runrev.com>
Sent: Friday, September 19, 2014 1:19 PM
Subject: Re: large arrays

> larry wrote:
>
> > As I mentioned, it choked starting with Rev 3.5 and every iteration
> > of Rev or LC that I've tried since, up to the current 6.1.1 (rc 4)
>
> The current shipping version is 6.6.3, and the latest test version lower 
> than 7 is 6.7RC2.
>
> Please let us know if you see any significant change with either version.
>
>
> > I have a hidden stack with my word fields in it - for words in length
> > from 2 to 12.  The word lists are optimized with a key for faster
> > searching.
> > When the program loads, it puts each of the 11 word list fields into
> > one array - which takes about 1 second on my XP with 4gb of ram.
>
> If the problem is the time it takes to load the array, and the loading is 
> happening from fields, the problem may not be in the arrays and perhaps in 
> the field accesses.
>
>
> > The word lists vary in length (3-ltr words is 1292 lines & the
> > largest, 9-ltr words, is 40,727 lines.)  Each line in all the lists
> > averages less than 100 characters.
> > The size of the field for 9-ltr words is 796kb if copied into a txt
> > document.
>
> So there are 11 elements in the array?
>
> If you run the array through arrayEncode and obtain its length, what is 
> the size reported?
>
>    put len(arrayEncode(tSomeArray))
>
>
> > If I put in a 9 letter word such as ELEPHANTS, the program will find
> > all anagrams (words found within the letters of ELEPHANTS) in about
> > 4/10 of a second.
>
> Is that the old time or the new time?  If new, what is the old time (or 
> vice versa)?
>
>
> > I spoke to someone at RunRev years ago and they acknowledged that
> > there was an issue with arrays.
>
> It would be interesting to know exactly what they'd said.  While arrays 
> have undergone revision a few times over the years, they tend to be very 
> fast in every version I've used.
>
> Hopefully once we understand more of the details of your code we'll pin 
> down the bottleneck and get your performance back on par.
>
> -- 
>  Richard Gaskin
>  Fourth World Systems
>  Software Design and Development for the Desktop, Mobile, and the Web
>  ____________________________________________________________________
>  Ambassador at FourthWorld.com                http://www.FourthWorld.com
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your 
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode 