LC7 and 8 - Non responsive processing large text files

[Roland Huettmann]

Thanks a lot for replies Mark and Monte and for everyone )

I like to stress that this is planned to be a version for a real product to
everyone out there... working with large files.

The anomaly starts immediately after calling the handler whether or not an
individual block of data is parsed.

HERE IS THE CODE SNIPPET :(shortened)

## File myFile is a very large text file containing messages including
encoded images etc. > 20 GB
...
// Opening the large file
open file myFile for binary read

put CR & XYZ" into tString
put 999999 into n

// Reading from file
...repeat n
......read from file myFile until tString
......put the result into tResult
......add 1 to i
...... if i mod 1000 is 0 then
.........if the commandkey is down then exit repeat
.........-- showStatus i,s -- separate handler
.........wait .001 sec
.......end if

// I disabled parsing for testing purpose
.......-- if tparse is true then
.........-- put parseMsg ( it ) into a -- separate handler to parse
....... -- put a["Sub"] & tab & a["Fro"] & tab & a["Da2"] & tab after
gData["lst"]
......-- end if

...... if tResult = "eof" then exit repeat
... end repeat

close file f
...
function parseMsg tMsg
--- do some processing and return parsed
end parseMsg

In LC 7 or 8 (not yet tested in 6, but why to test there since it is end of
life soon...?) on very larges files this makes LiveCode become
un-responsive..

If I set n = 3, or 10, or 20 or some small number then it also turns LC
into non-responsive state but  after processing LC becomes responsive again.

Setting n to 1000 or more for example it becomes unresponsive for such a
long time that I kill the application using Task Manager.

Also there is no way using the command key as defined in the script to stop
the process after 1000 iterations since LC is unresponsive.

So, in my opinion, the problem has nothing to do with parsing, it has to do
with reading the file. Any parsing and further processing will slow down,
but should not result with anything unexpected.

I will try to reinstall latest version 6 to see how that behaves.

LC should not become unresponsive in any case. Right? Or do I miss
something? Something wrong with the script?

And a more generalized question also discussed before: What exactly happens
when LC tries to read from a very very large file? Maybe it is Gigabyte or
even Terabyte file? It could just be too big to read and it should then
still not return empty or become unresponsive, but return some error
message.

And what happens when very large amounts of data are read into memory,
processed there, and placed into a field? Is there anything preventing
unresponsiveness?

Thanks again for feedback... )))) VERY MUCH APPRECIATED.

Roland




On 14 April 2016 at 07:55, Monte Goulding <monte at appisle.net> wrote:

>
> > On 14 Apr 2016, at 3:25 PM, Mark Talluto <mark at canelasoftware.com>
> wrote:
> >
> >> But reading something into memory using "read from file <filename> UNTIL
> >> <string>" and doing this many times over in such large text file creates
> >> non-responsiveness of LC (tested on 7 and 8).
>
> It would be interesting to know if the engine were doing lots of encoding
> conversions in order to test for the string. I must have missed the first
> email but I’d be interested to know if the file was opened for text or
> binary read and if an encoding was provided in the command if it was text.
> If it was binary is performance any better if the until <string> a variable
> with the same encoding as the binary file? The other thing worth pointing
> out is if you are doing a text read on an 8 bit encoding the size of your
> data doubles once read because each character is represented by 16 bits.
> When you consider the amount of data we are talking about here that will
> cause a serious slowdown.
>
> Cheers
>
> Monte
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>