Feature request: semaphores for externals

[Mark Wilcox]

I think that asynchronous and concurrent processing is going to be essential for LiveCode to remain relevant longer term. There is an irreversible trend towards more cores and greater parallelism in almost all environments.

In that context I think externals is too narrow a scope for adding more asynchronous operations to LiveCode, although it might be a good place to start. :)

Ideally I think some kind of closure, like blocks in Objective-C would be the way to go, with options for dispatching asynchronously or specifically to background threads - externals called from within that closure could be synchronous or asynchronous, they simply complete the operation when they're done - LiveCode will already be doing this in async mode in either case.

Specific to your proposal, if the semaphores are to be created by the external then possibly a semaphore itself is unnecessarily over-specified - unless you were planning to contribute an implementation, in which case the engine contributors board is probably the best place to continue the discussion. At a minimum there needs to be *a* way for externals to trigger, or even better call (with parameters), a script that will be run on the main thread at the next opportunity - what mechanism the external uses to handle the asynchronous processing in the background could well be platform-specific or vary case by case.

If you weren't referring to OS level semaphores and just the abstract concept as something to add to LiveCode for signalling completion of async events then I think it seems a bit of a low-level construct for a language as high-level as LiveCode - if you consider the case where you have multiple async calls outstanding you're basically adding something like an OS scheduler into the event loop. How would this look to callers of the external from within LiveCode and how do you ensure any variable the external call will update is not being accessed in parallel? It's issues like that which make me suggest something like blocks.

Mark


________________________________
 From: Paul D. DeRocco <pderocco at ix.netcom.com>
To: LiveCode list <use-livecode at lists.runrev.com> 
Sent: Sunday, 21 April 2013, 5:44
Subject: Feature request: semaphores for externals
 
One of the long-bemoaned limitations of externals is their inability to send
messages into the LC environment asynchronously. I can imagine why this
might be very difficult to implement, given that externals often would like
to do this from within the context of a different thread, or even in an
operating system callback that is called on some unknown thread. I came up
against this in writing a MIDI external: output was trivial, but the only
way to handle input was to dump it into a FIFO and then poll it from LC in a
timer message handler.

The timer message is already one mechanism in LC that allows an asynchronous
event to cause a message to be sent: "send <message> to <whom> in <time>".
I'm not sure how this is implemented, but the key design point is that the
message is prepared in advance, by the LC program, not generated by the
arrival of the time, so all the time signal has to do is wake up LC if it is
in a waiting state, and set some flag telling LC to dispatch the existing
message at the earliest valid point in its message processing loop. A
similar mechanism could be implemented for the benefit of externals using
semaphores, where the setting of the semaphore would wake up LC if it is in
its waiting state.

Since a semaphore would only be useful in the context of an external, it is
the external that should create the semaphore, typically in its
initialization function. LC should provide a function that does this, and
that returns a handle of some sort which the external can then return to the
LC program in some manner, perhaps by storing it in a LC variable provided
as a parameter to the initialization function. And of course, LC would have
to define a function that the external could use to set the semaphore.

The LC program could then say something like "send <message> to <whom> on
<semaphore>". If the semaphore is already set, it would be cleared and the
message would be sent immediately; if the semaphore was clear, setting it
would wake up LC if it is in its waiting state, and in any case cause the
message to be dispatched at the earliest valid time, leaving the semaphore
cleared. Typically, the message handler would then call some other function
provided by the external to deal with the event.

Ideally, the clearing of the semaphore would occur at the point in time when
the message is dispatched. The semaphore might actually be implemented with
some other OS object, such as a signal in Linux.

One might extend the syntax a little further, allowing "send <message> to
<whom> on <semaphore> in <time>", to specify a time limit. On entry to the
message handler, "the result" could have a value of "true" if the semaphore
had been set or "false" if the time limit had expired.

So LC would only need to have three functions added to its externals API: a
function to create a semaphore, another to delete it (rarely needed), and a
function to set the semaphore which can be called from any context. Also,
"pendingMessage" would have to add a fifth item, the semaphore handle, to
each line. Either the time or the semaphore could be empty.

Comments are welcome.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco at ix.netcom.com 



_______________________________________________
use-livecode mailing list
use-livecode at lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode