LC7 and 8 - Non responsive processing large text files

Roland Huettmann roland.huettmann at gmail.com
Fri Apr 15 02:59:57 EDT 2016


< Richard: "You may find the sample stack attached to this report useful:
<http://quality.livecode.com/show_bug.cgi?id=17210>"

Richard, thanks a lot for the sample stack. I downloaded and tested
"LargeFileRead.livecode". Unfortunately it also creates the same symptoms
on Windows:

Stack becomes unresponsive in Windows (using Windows 10). As I see, you are
also using both "read until..:" and "read for..." in two different loops.
Also here the problem is with the "read until..:" loop.

But then I added the recommended "wait 0 with messages" to the "read until
" and ...

.....................THAT WORKED! ......................... )))))))

Though it does NOT work for me as I am reading much larger chunks of data
using "read until..."

(It is still running for 60 minutes now... I think I have to force-stop it.
No... now it finished with 3,060,000. -:)

I think it is important to know how to handle "big" data with files, in
memory and placing in fields. And of course in a future nearby 8.1 version
(or sooner?) those in need would love to see this taken care of in the
engine.

TESTING READING LARGE GIGABYTE FILES

To make a contribution here, I am going to test all the variants and
techniques discussed, published by you or developed by me, and creating a
test case checking for performance and stability using the different
methods as suggested. It will take some time during the weekend, or so.

It will be interesting to see ...

Roland









On 14 April 2016 at 20:47, J. Landman Gay <jacque at hyperactivesw.com> wrote:

> On 4/14/2016 12:27 PM, Roland Huettmann wrote:
>
>> There is a work-around for me:
>>
>> Not using "read until <string>" but reading "at" <position> "for" <number
>> of bytes" also in this huge file.
>>
>> I can read even 100 MB into memory (it does not create a big problem) and
>> then process using offset() and then reading the next pack of data.
>>
>
> As Mark suggested, that implies that the search string is uncommon in the
> text and large amounts of data need to be parsed before a match is found.
> By reading for a specified number of bytes instead, you guarantee a limited
> amount of parsing.
>
> --
> Jacqueline Landman Gay         |     jacque at hyperactivesw.com
> HyperActive Software           |     http://www.hyperactivesw.com
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



More information about the use-livecode mailing list