LC7 and 8 - Non responsive processing large text files

[Richard Gaskin]

Roland Huettmann wrote:

 > There is a work-around for me:
 >
 > Not using "read until <string>" but reading "at" <position> "for"
 > <number of bytes" also in this huge file.
 >
 > I can read even 100 MB into memory (it does not create a big problem)
 > and then process using offset() and then reading the next pack of
 > data.

You may find the sample stack attached to this report useful:
<http://quality.livecode.com/show_bug.cgi?id=17210>

It's similar to the buffering script Mark posted earlier, but uses a 
constant conveniently located at the top of the script to govern the 
buffer size.

The reason this may be useful is that when I was experimenting with 
buffer-and-parse-in-memory options like you're doing now I was surprised 
to find that smaller buffers can be MUCH faster than larger ones.

On my Linux box the optimal buffer turned out to be 128k, but I'd be 
very interested to learn which buffer size works best on your Windows 
system.


 > Now I am creating an index for each occurrence which is a position
 > for each block of data within the file. Well, that index alone is
 > larger than 100,000 lines. I am not sure here using an array as I
 > read too many times that they are very slow in LC8... ???

Performance differences will vary as much by algorithm as by LC version.

Whether an array is "slow" compared to an alternative will depend on 
what's being done with it.

I'd test an array version and an alternative before relying on 
discussions others have had about algos that may be quite different from 
your own.

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com