LC7 and 8 - Non responsive processing large text files

[Roland Huettmann]

> One thing to remember is as much as we would like it to be the engine is
not prescient.
> It does not > know how much data it will need to read before it will find
your sentinel string.
> What this means in > practice is when reading until a string the engine
needs to:
>  - constantly resize the memory buffer when it nears overflow for data
 > that is read in because it > doesn’t know how big it needs to be
>  read one byte at a time then check if the last N bytes match your
sentinel string

Monte and Mark. Exactly this is the point! I completely agree.

What I tried to say - maybe I am not expressing it so well - is that by
experience so far it is WORKING WELL READING VERY LARGE FILES just using
 "READ ... FOR" in a large file and then processing later in a separate
step.

The problem I had was when testing the "READ ... UNTIL" - and you commented
this in depth.

I already have been successful before building an index and then reading
just pieces of data using the index. No odd behaviour. No "non
responsiveness".

YES Mark, I already was splitting the process! And even before writing to
this list, I had done that intuitively in a similar way. It was working and
it is working. And I can only suggest to anybody interested to follow your
advise and study the scripts you and other provided.

>From all you said I get a picture which I did not have before in such
detail. And I hope others get it too. It makes us understand more and more.
So, all my thanks to your kind contributions.

I will follow up here...

Greetings, Roland













On 15 April 2016 at 10:05, Monte Goulding <monte at appisle.net> wrote:

>
> > On 15 Apr 2016, at 4:59 PM, Roland Huettmann <roland.huettmann at gmail.com>
> wrote:
> >
> > Though it does NOT work for me as I am reading much larger chunks of data
> > using "read until..."
> >
> > (It is still running for 60 minutes now... I think I have to force-stop
> it.
> > No... now it finished with 3,060,000. -:)
> >
> > I think it is important to know how to handle "big" data with files, in
> > memory and placing in fields. And of course in a future nearby 8.1
> version
> > (or sooner?) those in need would love to see this taken care of in the
> > engine.
>
> Roland are you ensuring to encode your sentinel string as the same
> encoding as the file? I’m interested to know how much difference that makes
> as I think this could be the main source of any difference you are seeing
> between LC 6 and 7/8 on this.
>
> One thing to remember is as much as we would like it to be the engine is
> not prescient. It does not know how much data it will need to read before
> it will find your sentinel string. What this means in practice is when
> reading until a string the engine needs to:
>  - constantly resize the memory buffer when it nears overflow for data
> that is read in because it doesn’t know how big it needs to be
>  - read one byte at a time then check if the last N bytes match your
> sentinel string
>
> Obviously this is significantly more expensive than read for because the
> buffer is allocated once and the read can read the number of bytes you want
> and give it to you straight away because it doesn’t need to check it
> anything matches.
>
> I second Mark’s recommendation to move the file parsing to a helper
> process if possible because no matter what this is going to be a bottleneck
> in your app.
>
> Cheers
>
> Monte
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>