Large Recordsets - An unbuffered approach?

[Lynch, Jonathan]

Well...

If he lacks the ability to perform analysis on the 60-million records on
the server side, then whatever solution he finds is going to be slow.

Perhaps he does not need a list of record IDs, just a range? A starting
and ending value could be enough.

Parallel processing can be accomplished with Rev, but it is complicated.
Basically the program needs to either spawn or activate a standalone
that will do all the analysis, and the standalone would either
communicate with the primary program via ports, or via the creation of a
result file. If he is going to have analysis running for a long period
of time, this sort of approach would be useful.

-----Original Message-----
From: use-revolution-bounces at lists.runrev.com
[mailto:use-revolution-bounces at lists.runrev.com] On Behalf Of Ruslan
Zasukhin
Sent: Thursday, December 08, 2005 2:39 AM
To: use-revolution
Subject: Re: Large Recordsets - An unbuffered approach?

On 12/8/05 12:30 AM, "Lynch, Jonathan" <bnz2 at cdc.gov> wrote:

> This is prolly an ignorant suggestion, but...
> 
> Rather than returning the entire record set, can you just return a
list
> of the record IDs from your query? 60-million record IDs should not
> overwhelm your memory.
> 
> Then you can go down the list, and for each record (or a predetermined
> number of records), you load the content, perform your analysis on
each
> record, then discard that Information.

    60M * 4 bytes = 240 MB

Not so small.
And if this will happens via network -- exactly not fast.

-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]


_______________________________________________
use-revolution mailing list
use-revolution at lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution