What to expect when memory's running low

[Gregory Lypny]

Hello everyone,

I'm working on a stack that indexes big (15 to 200 MB) text files and 
breaks them into smaller files with similar information.  This is done 
by reading the source file into an array whose elements get written to 
their own output file.  The output files are tab-delimited text, 
between 500 KB and 3 MB, and typically have 100,000 lines (each line 
represents  information from a variable or field from one record of the 
source file, so you can think of the source file as being a database 
with 100,000 records).

My stack works fine (surprisingly so for me...) as long as I don't mess 
with more than 90,000 records,  I can hardcode a read of about 90,000 
lines into various array elements and then dump them to their own 
files.  However, when I let the stack run through the entire source 
file of about 104,000 records (letting it read until an eof is 
reached), it hangs (spinning coloured beach ball in Mac OS X) when 
writing the very last array element to its own file.  If I force quit 
MetaCard, and check the individual output files, it turns out that all 
of the data is, in fact, written successfully, so the main handler is 
doing its job, but there isn't enough juice to refresh the stack window 
(for instance, to display a status field that indicates that the 
process has been completed, the time, number of records, etc.) or give 
control back to MC (beach ball to pointer).  I've tried to minimize 
memory usage by putting empty into array elements or deleting them 
entirely once they are no longer needed.

I guess my question (sorry for the diatribe) is this:  Am I dealing 
with a memory problem?  And, if so, what's the best way around it?  I'm 
considering writing to files repeatedly rather than once at the end, or 
running the handler two or more times, effectively breaking up the 
reading of the source files into many parts.

	Greg

	Happy X-Mas



Gregory Lypny
Associate Professor
John Molson School of Business
Concordia University
_________________________
"Absence of evidence is not evidence of absence."
                 - Anonymous