Making Revolution faster with really big arrays

[Dennis Brown]

To Alex & Dick,

Yes, I am doing something like what Dick suggested, only a lot more 
complex --actually I have about a hundred very different calculations 
to do on various arrays which is why I was not more specific about the 
calculations (it has to do with generating various statistics and 
indicators).  Your suggestion using the split is just the ticket to 
solve my problem in leu of an improved transcript --which would still  
be 15 times faster for each pass, but it sure beats all the other ways 
I've come up with.  Please accept my humble thanks for all your help.

To Frank & Mark & Brian & Robert & Pierre & all,

I have been overwhelmed by all the various offers of help in a few 
short hours from posting my problem.  You are a wonderful community of 
helpful people.  Thank you all very much.

 From the content of the replies, and the complexity of some of the 
proposed solutions it seems more obvious to me now than before that an 
improved sequential access method for data needs to be included in 
Transcript.  After all one should not have to jump through hoops to do 
such a simple and common task.  I guess the shortcoming  of the random 
access methods used in the language are not apparent until it is used 
with really big arrays which amplify the speed issue.   From my 
previous posted timings, you can see how much faster sequential access 
is than any of the other methods.  I can't accept the answer that the 
language is unsuitable for a task, because it is interpreted, when a 
minor improvement would make it suitable.  Let's fix the problem 
instead of thinking that Revolution can not be used for such tasks.  I 
will bet improved sequential access would find its way into most 
projects.

I look forward to more discussions about number crunching improvements,

Thanks,
Dennis

On Apr 13, 2005, at 9:59 AM, Alex Tweedly wrote:

> Dennis,
>
> could you post the "dumb, too-slow" code to show what you need to do ?
> The more specifically we can see the problem, the more "imaginative" 
> solutions we might come up with.
>
> I suspect some variant of Dick's suggestion (using split on 
> all-but-one of the lists, and "for each"-ing over that remaining one) 
> - perhaps repeated internally on the items of each line - will get you 
> an adequate solution --- but we'll get there more quickly and 
> certainly if we know more closely what we are trying to do.
>
> -- 
> Alex Tweedly       http://www.tweedly.net

On Apr 13, 2005, at 5:41 AM, Dick Kriesel wrote:

> On 4/12/05 7:36 PM, "Dennis Brown" <see3d at writeme.com> wrote:
>
>> ...I have two arrays 10,000 lines with 2500 items in each line...
>> ...I need to pair the items together...
>
> Do you have something like closing prices in one container and shares 
> traded
> in another container, each for the same 10,000 securities over the same
> 2,500 trading days?  If so, do you want to calculate something like 
> trading
> volume as price * shares?
>
> If so, then here's a way to bring together the values from one such 
> list
> with the values from another, without waiting for any chunk 
> expressions.  On
> my machine, generating a 10,000 line by 2,500 item list of integers to
> prepare for the test took about three minutes.  Then bringing together 
> all
> the pairs of lines from two lists, as you indicated in your initial 
> email,
> took about 1.2 seconds.
>
> So you could do many passes through your big lists before the day is 
> done,
> and not need to wait for an extension to the language.
>
> To try it, paste the following into the script of a new stack.  If I've
> misunderstood or done something wrong, please let me know.
>
> -- Dick
> ---------------------
> global gList
>
> on mouseUp
>   if gList is empty then initialize
>   combinePairsOfLines gList,gList
> end mouseUp
>
> on initialize
>   put 10000 into tLineCount
>   put 2500 into tItemCount
>   put tLineCount && tItemCount & cr
>   put the seconds into tSeconds
>   repeat with i = 1 to tLineCount
>     repeat with j = 1 to tItemCount
>       put i + j & comma after tList
>     end repeat
>     put return after tList
>     if i mod 1000 = 0 then put "line count so far:" && i & cr after msg
>   end repeat
>   put tList into gList -- save for reuse across edit-test cycles
>   put "elapsed seconds to initialize:" && \
>       the seconds - tSeconds & cr after msg
> end initialize
>
> on combinePairsOfLines pList1,pList2
>   put the long seconds into tSeconds
>   split pList2 using return
>   put 0 into i
>   repeat for each line tLineFromList1 in pList1
>     add 1 to i
>     put tLineFromList1 & pList2[i] into tCombinedLine
>     -- doSomethingWithCombinedLine tCombinedLine
>   end repeat
>   put "elapsed seconds to combine pairs:" && \
>       the long seconds - tSeconds & cr after msg
> end combinePairsOfLines
>
>