LC7 and 8 - Non responsive processing large text files

[Roland Huettmann]

This issue was addressed before within another context, but I am running
into the same problems with all versions of 7 and 8 including the latest
rc1 (using Windows 8.1 to Windows 10).

The question is important as it is related to a planned project for very
serious development and a product to be developed with LiveCode 8.0 (stable
version).

Imagine Gibabytes big text files. Maybe even it could be Terabytes!

NON-RESPONSIVENESS USING "UNTIL" READ

I found opening a very large text file (my file is 26 GB), simply reading
from it up to 90 MB of data in each iteration, using an offset () function
processing the read junk of data, and iterating through the file, is not a
problem and performs acceptably.

But reading something into memory using "read from file <filename> UNTIL
<string>" and doing this many times over in such large text file creates
non-responsiveness of LC (tested on 7 and 8).

So, what do I do?

1. Open file in read only mode and for binary read
2. Reading until <string> (which separates pieces of data where each piece
is not bigger than 30 MB)
3. Repeating the reading 10x or 100,000x or more... (depends on file size)
4. Performing some additional processing on such read pieces of data in
memory
5. Eventually placing data into a field, or storing it away somewhere else
(max 30 MB in my case)
6. Closing the file

LC eventually turns back to normal state when user is waiting long enough
and the process is finished. But LC is completely locked away in such
non-responsive state, and it is not acceptable to our project.

Maybe with each loop I have to send a waiting message? I tried sending with
10 milliseconds, but nothing changed. I tried several other options.

One more note: Additionally I find that it can take significant delays when
placing large blocks of text into a field. Is there an idea of improving
this?

Kind greetings
Roland