LC7 and 8 - Non responsive processing large text files

[Roland Huettmann]

Hi all, great response. Enjoyable to read, and motivating a lot .... ))).

> Mark: ... "When you say 'non-responsiveness' I take it you mean that
Windows thinks that the application has 'hung'? (i.e. the windows go
slightly opaque)."

That is correct, Mark.

As I understand it now, it is Windows making the application to be
unresponsive. To be more exact in describing: Immediately after the
function is called (why "immediately"?), the cursor changes to a spinning
blue wheel running, the LC windows title is suffixed with "(Not
Responding)" and the stack window appears a opaque.

> Paul: ... Using "wait 0 with messages" ...

I did that. Unfortunately it does not help much. But there is a change: Now
it switches from responsive to non-responsive state in every loop. So there
are responsive phases now, and unresponsive phases. ) But I still can not
stop the loop using the command key.

Since for testing I am not doing anything but using "read until <string>"
within the loop, not touching the read data, it seems obvious to me (...?)
that is has to do with the "until <string>" inner processing.

Yes, it is a "binary read".

> Mark: ... However, you might find replacing 'read until' with a script
solution might make things work better: ..."

That is something to really test and I will do this. Looks sophisticated !)

=== Workaround ===

There is a work-around for me:

Not using "read until <string>" but reading "at" <position> "for" <number
of bytes" also in this huge file.

I can read even 100 MB into memory (it does not create a big problem) and
then process using offset() and then reading the next pack of data.

Now I am creating an index for each occurrence which is a position for each
block of data within the file. Well, that index alone is larger than
100,000 lines. I am not sure here using an array as I read too many times
that they are very slow in LC8... ??? Well, it is better for access. Or
what is the best way of creating a very large index and where best to
store? As a text file? Or using local SQLite?

In case of reading from a file position for a defined number of bytes there
is no strange behaviour as far as I experienced until now. It takes a
little while, but acceptable. All is responsive.

In other words, reading a large file "from <startPosition> for
<numberOfBytes> seems ok even in a loop where it is still advisable to use
"wait 0 with messages".

Well - better now to test and test and test...

Continuing to explore this. It is needed.

Cheers, Roland





On 14 April 2016 at 16:41, Richard Gaskin <ambassador at fourthworld.com>
wrote:

> Mark Talluto wrote:
>
> > Version 7/8 are slower than version 6 in almost every way. Here are a
> > couple of bug reports on the issue:
> >
> > <http://quality.livecode.com/show_bug.cgi?id=16387>
> > <http://quality.livecode.com/show_bug.cgi?id=15711>
> >
>
> You may want to add this enhancement request to that list:
> <http://quality.livecode.com/show_bug.cgi?id=17210>
>
> It deals specifically with reading/parsing large files, using two methods:
> the common "read until <char> form, and one using a custom buffer similar
> to Mark's email earlier today.  Both forms use binary mode, so encodings
> don't come into play.
>
>
> > There are others out there. The team has been trying to make LC 8
> > more performant though. So our reports are being worked on.
>
> I discussed the performance concerns with Peter in our call last week. As
> you noted, they've been working on them, and in some cases have had
> significant success.
>
> Some chunk expressions, esp. related to lineoffset, and now nearly as fast
> as in v6.x, while also retaining all the features Unicode now offers.
>
> Other forms of chunk expressions may benefit from further performance
> enhancement, and similarly with arrays some performance gain have been made
> in in recent versions while others are still on the to-do list.
>
> File I/O remains an area slated for review in an upcoming version, and I
> haven't yet tested socket I/O so I don't know if performance there has
> changed but is obviously just as important.
>
> Given that v8.0 is now in RC, it would seem extremely unlikely that any
> enhancements not already close to completion will land in v8.0 Stable.
>
> That said, the team seems eager to increase performance as much as
> practical, and with the progress made thus far -- with v8 being much faster
> in many operations than v7 -- I'm confident we'll see a version after v8.0
> that will continue this trend.
>
> --
>  Richard Gaskin
>  Fourth World Systems
>  Software Design and Development for the Desktop, Mobile, and the Web
>  ____________________________________________________________________
>  Ambassador at FourthWorld.com                http://www.FourthWorld.com
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>