LC7 and 8 - Non responsive processing large text files

[Mark Waddingham]

On 2016-04-15 08:59, Roland Huettmann wrote:
> I think it is important to know how to handle "big" data with files, in
> memory and placing in fields. And of course in a future nearby 8.1 
> version
> (or sooner?) those in need would love to see this taken care of in the
> engine.

I don't think the problem here is whether the engine has the facilities 
to process large files - it clearly does given the number of examples of 
how to do it that have been provided ;)

The issue here is how to keep a UI responsive on the Desktop whilst 
running a long processing operation - this isn't a problem unique to 
LiveCode it is one which pertinent to all languages on all OSes.

 From what I can understand of your use-case you are wanting to index an 
*exceptionally large* file so that you can look up things from it at a 
later date. Indeed, it seems to me that the problem you are trying to 
solve falls into two pieces:

   1) Process the very large file and produce an index of 'points of 
interest' in the file.

   2) Subsequently use the index to perform the main function of your 
application.

With this in mind, I'd suggest considering separating these two parts 
and run them in separate processes.
The foreground (UI) application can launch the second process to 
generate the index and provide a simple progress update periodically 
without having any impact on the performance of its job, and then when 
that operation is completed the foreground application can load the 
index to use.

The LiveCode Installer (albeit no longer used on Mac) works like this. 
It is a single application which can run in two modes. In the foreground 
mode you see the UI which you interact with, but when you finally get to 
the 'install' button, when you click that it launches a background 
process which runs a installation script - it feeds back progress 
information back to the foreground process at suitable intervals so that 
the UI can update a progress bar. (Admittedly the installer *has* to 
work this way as the background 'slave' process has to run with 
administrator rights, however it also happily solves the UI progress bar 
update / cancellation problem though too).

This is basically what you could call a 'master/slave' pattern. You have 
a master process which provides the UI and the application's main 
function, and a slave process which is run and controlled by the master 
during indexing operations. This model has all kinds of advantages - for 
example, you can rerun the slave 'indexing' process whilst the user is 
still using an existing index; or indeed launch several slave processes 
to index multiple files at once.

In LiveCode such a pattern can be implemented in the master using an 
'open process read/write process close process' loop; whilst all the 
slave has to do is read from stdin and write to stdout.

The above might seem a little opaque - however, it might help to have a 
look at:

   
https://github.com/livecode/livecode/blob/develop/builder/installer_utilities.livecodescript

Which is the main installer implementation. When run as slave, the only 
handler which runs is 'runInstallerActions'; and when run as master 
there is a UI which runs a 'send in time' loop 'installerMonitor' to 
talk to the slave. The main entry point to the installer application is 
a startup handler which decides (by looking at the command line 
arguments) what it should do:

on startup
    local tAction

    if $1 is "install" then
       put "install" into tAction
    else if $1 is "doinstall" then
       put "doinstall" into tAction
    else if $1 is "uninstall" then
       put "uninstall" into tAction
    else if $1 is "douninstall" then
       put "douninstall" into tAction
    else
       set the itemDelimiter to slash
       if the last item of $0 contains "setup" then
          put "uninstall" into tAction
       else
          put "install" into tAction
       end if
    end if

    switch tAction
       case "install"
          if $2 is "noui" then
             runFacelessInstall
          else
             hide me
             send "runInstallerUI" to me in 0 millisecs
          end if
          break
       case "uninstall"
          if $2 is "noui" then
             runFacelessUninstall
          else
             hide me
             send "runUninstallerUI" to me in 0 millisecs
          end if
          break
       case "doinstall"
          runInstallerActions
          break
       case "douninstall"
          runUninstallerActions
          break
       default
          quit 1
    end switch
end startup

Here 'install' and 'uninstall' run the master UI (the default is 
'install' if no arguments are specified); whilst doinstall/douninstall 
run the slave.

Anyway, I'll try to find some time to distill out a skeleton application 
which does this - it seems like it might be useful thing as an example 
:)

Warmest Regards,

Mark.

P.S. One could argue that 'multi-threading' might help here - but it is 
actually absolutely no different in effect from splitting into two 
processes. Indeed, for this kind of thing separate processes is a much 
much better idea as it completely insulates the master from the slave 
and is much more failsafe as the two processes run in their own memory 
spaces with their own resources and only communicate over a very thin 
connection. i.e. Problems in one cannot propagate to the other. It also 
provides more options in terms of running the indexing operation - it 
doesn't necessarily need to be done in response to the master 
application, indexing becomes an operation in its own right which can be 
managed in any way which might be required.

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps