Faceless MC -- Memory Failure

[Brian Yennie]

Sadhu,

I would suggest just defining a constant for how many lines to read at 
once, and read from the files in chunks. Alternatively, you could pass 
the number of lines to read at once in $2. I really don't think you 
want to be reading the entire file at once if there's any chance of 
800MB or larger files- that'll eat up gigabytes of RAM by the time you 
process it...

I would try looking at one of your sample files, and figure out about 
how many lines make up 1MB, and then extrapolate from there how many 
lines you'd like to process based on how much memory you want to use.

EX:

constant kNumberOfLines = 20000
on startup
    put $1 into inputFile
    open file inputFile
    repeat
       read from file inputFile for kNumberOfLines lines
       if (it is empty) then exit repeat
      processData it
    end repeat
   close file inputFile
end startup

on processData inputData
   // find reversed checks
   repeat for each line thisLine in inputData
     if (char 1 to 2 of thisLine contains "PC") then
       put char 4 to 11 of thisLine \ -- reversed by ck seq #
         & "|" \
         & char 13 to 22 of thisLine \ -- check effective date
         & linefeed after pcDateArray
     end if -- a reversed check
   end repeat

   split pcDateArray by linefeed and "|"

   // patch the date of the reversing check
   repeat for each line thisLine in inputData
     if (char 1 to 2 of thisLine contains "PR") then
         put char 24 to 31 of thisLine into sequenceNumber
         put pcDateArray[sequenceNumber] into char 13 to 22 of thisLine
     end if -- reversing check

   // now pass the modified data on
     put thisLine
   end repeat
end processData