Squishing data

[xbury.cs at clearstream.com]

Shari,

The biggest stack I ever made was around 70MBs with 7 levels of data all 
interelated.

The key to speed (and this was HC!) was to link the data to the right cd 
id and then grab
the data from there. Access times were less than 3 seconds - it took 3 
days to build or
update the links though! 

Size was not important as it was for a CD but despite cd speeds (way back 
then it was 4X), 
was really snappy!

The scheme was a display stack with browsing controls and 7 hidden stacks. 
The 1st level
stack contained the index to the cd IDs holding the data and their 
respective parent and child
links (also stored as IDs). IOW, a classic double-linked list. 

It didn't require much ram and it was the fastest I could find in the 
year's time it took to develop.
I had planned using an XCMD in the beginning but none could handle the 
amount of data! 
Even FileMaker had troubles! There were some 800,000 record links for 
80000 items!

Hope that helps a bit! ;))
Xavier

---------------------=---------------------
Xavier Bury
TNS NT LAN Server
ext 6465




Brian Yennie <briany at qldlearning.com>
Sent by: metacard-admin at lists.runrev.com
28/08/03 02:20
Please respond to metacard

 
        To:     metacard at lists.runrev.com
        cc:     ^
        Subject:        Re: Squishing data



I'm not sure what you mean by "putting data in an array" changing the 
stack size: arrays are held in memory. Do you mean that you write the 
arrays to custom properties?

You may want to try using compress() on the field data, i.e.
set the compressed_data of fld 1 to compress(fld 1)

Then just access the fields with:
put decompress(the compressed_data of fld 1) into fieldData

For performance, you would want to only decompress() once- although 
keeping the uncompressed data in memory bumps your memory requirements 
so you'll have to balance memory vs. performance.

Compress() should get you a reasonable shrink on text, hopefully 50% or 
so.

Brian

> The stack with 300 fields is about 15 MB.  I tried putting the data 
> into an array, and deleting the fields.  But it actually makes the 
> stack bigger to do this.  I created an array with 300 elements, 
> myArray[1] held the data from field 1, and so forth.

_______________________________________________
metacard mailing list
metacard at lists.runrev.com
http://lists.runrev.com/mailman/listinfo/metacard




Visit us at http://www.clearstream.com
                                                          
IMPORTANT MESSAGE

Internet communications are not secure and therefore Clearstream International does not accept legal responsibility for the contents of this message.

The information contained in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any views expressed in this e-mail are those of the individual sender, except where the sender specifically states them to be the views of Clearstream International or of any of its affiliates or subsidiaries.

END OF DISCLAIMER
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.runrev.com/pipermail/metacard/attachments/20030828/f904a68f/attachment.htm