Large Recordsets - An unbuffered approach?

Ruslan Zasukhin sunshine at public.kherson.ua
Wed Dec 7 15:56:24 CST 2005


On 12/7/05 11:34 PM, "Adam" <friendlygeek at gmail.com> wrote:

> Hi All:
> 
> My project works. Sort of. ;)  Thanks again to those who have been
> helping me out.  I have done in two days what took me weeks to formulate
> elsewhere.  Best of all it has a GUI. :D
> 
> Like I said it sort of works... I can connect to the database, I can
> retrieve records, but this is the sorce of my problem.  It would appear
> to me that an attempt is made to retrieve all the records matching the
> SQL criteria specified in the query before opertating on the contents of
> the recordset.  With a small recordset this approach in fine and dandy.
>   But in my case, my criteria returns 60 million plus, records.
> 
> Previously, I had used PHP's mysql_unbuffered_query to fetch one record
> from the database at a time which I could then process and discard,
> moving to the next record.  I only had to execute one query to
> accomplish this but I would have to loop through a while statement to
> the effect of:
> 
> psuedo-code
> --
> 
> While lRecordCount <= lNoOfRecords - 1
> Fetch the current record
> Process the current record
> Increment lRecordCount
> End While
> 
> This worked very fast as it didn't have to move the entire result set to
> memory. (Which my laptop ran out of this afternoon) ;)
> 
> An aside:  Is it possible to have multiple threads?  Whereby an SQL
> query such as that listed above can operate while still allowing access
> to the program interface?

Hi Adam,

You have meet 2 problems:

* too big result
* mySQL have only client-side cursors, so it try to load result into RAM
into RAM of client computer.

You can try to use LIMIT, but often this is not what you want.


-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]





More information about the use-livecode mailing list