Memory-saving strategies

[Gregory Lypny]

Hello everyone,

I've created a couple of simple (and primitive) utility stacks that 
index big text files (15 MB to 220 MB).  These files are essentially 
the contents of flattened relational databases that have a descriptor 
at the beginning of each line followed by corresponding data.  There 
can be many instances of the same descriptor on different lines (that's 
the flattening part).  Scientific data is sometimes made available in 
this format at university FTP sites.  My utility has a main handler 
that creates arrays indexed by the descriptors and dumps the contents 
of each into a field on its own card for viewing, searching, exporting 
or merging with other data sets.  The fields holding the data often 
have more than 100,000 lines.  I'm also playing with writing the array 
elements to individual text files and then displaying them in card 
fields on the fly.

When messing with so much data, I've been forced to find ways to save 
memory so that the indexing handler does not bog down and so that 
navigating from one card to another in the resulting index stack is 
reasonably smooth.  (To be conservative, I do most of the development 
on a 300 mHz iBook running OS X with 192 MB of RAM.)  Not being an MC 
expert, I've approached the memory issue intuitively, and I'm wondering 
whether the things I've been doing really do help save memory and 
whether there are other things I should be doing.  Here are some of 
them.

- close the index stack and empty out the fields before creating a new 
index
- delete array variables as soon as they've done their job in the 
indexing handler
- put empty or zero into array variables to reinitialize them if they 
are reused in a repeat-with loop
- compact all stacks before saving them

I'd appreciate any memory-saving tips that you'd be willing to share.  
I've received so many invaluable suggestions from this list.  I'll have 
a generic version of one of the stacks ready in about a week (slow, I 
know, I have a day job), and I'd be happy to share it (perhaps it would 
make an interesting autopsy for those who want to get into the head of 
a truly novice scripter); you're welcome to contact me off list.

	Regards,

		Greg



Gregory Lypny
Associate Professor
John Molson School of Business
Concordia University
_________________________
"I am just going outside and may be some time."
    - Captain Lawrence Oates, 17 March 1912, Scott Expedition