Remotes - the new, exciting alternative to Externals.

Alex Tweedly alex at tweedly.net
Sat Jul 9 17:28:42 EDT 2005


I've been playing with writing externals for Revolution lately. It's 
been interesting. I'm very glad to know I can do it, and should I ever 
need to, I will. But I can't say it's been a lot of fun. Writing C code 
is frustrating and not very productive in the best of circumstances 
(e.g. supported by a symbolic debugger, protected processes, etc.) In 
the circumstance of running an external with Rev's IDE (where many 
errors cause immediate termination of the IDE, with no diagnostic info 
provided at all, far less any chance to debug), it is very unproductive. 
So while it's good to know that externals are possible for me, I also 
know that they are to be avoided if reasonable alternatives exist.

So I started looking for alternatives .....

Reasons to write externals in the first place.
1. to get access to hardware or other facilities not otherwise available
2. to get higher performance (e.g. matrix ops, image processing, etc.)
3. to take advantage of existing library packages
4. other significant but less common reasons

As described above, I don't like C (far less C++). The language I do 
like (assuming Transcript isn't an option) is Python.

Python is, in many ways, similar to Transcript. They're both scripting 
language, dynamic types (or typeless), associative arrays, etc. - that's 
largely why I like them both.  But they're also quite different, and 
complementary, in many ways. So, without falling into the trap of 
becoming a comparison between the two languages, I believe that a number 
of the reasons for writing an external would equally apply to using Python.

Of course, it would be difficult to write an actual external in Python. 
But I realized that one of the strengths of both Python and Transcript 
is that they have good, simple socket facilities. So I decided to try 
"Remotes" - socket-based Python add-ons to Transcript.

I wrote a small (30 lines) Python server which accepts "commands" over a 
TCP socket, and returns its result over the same TCP connection  (see 
the end of the email for the source code of the Python program). The 
"commands" are in fact snippets of Python code; so I can develop the 
Transcript stack and just change the snippets of Python to be sent and 
executed, without even restarting the server. So the whole process is as 
dynamic as Rev usually is.

As an example, I used the Python Imaging Library to calculate the 
Histogram of a JPEG, done as two steps (to avoid making it too easy). So 
there are two buttons - one to open and specify a file, the other to 
calculate th histogram for it.
The script of the first button is

> on mouseUp
>   local t
>   answer file empty with filter "JPEGs,*.jpg"
>   put it into theFile
>   put "import PIL.Image" & cr \
>       & "from PIL import *" & cr \
>       & "global im" & cr \
>       & "im = Image.open(" & quote & theFile & quote & ")" & cr  into 
> theScript
>   put theScript & cr & "endofscript" & cr into t
>   reStart
>  
>   SendPacket t
> end mouseUp

While the second is

> on mouseUp
>   local t
>   reStart
>   put "hi = im.histogram()" & cr \
>       & "return_result_list(wfile, hi)" into theScript
>   put theScript & cr & "endofscript" & cr into t
>   SendPacket t
> end mouseUp

Notes.
1. The "restart" handler is used to reset any open sockets, then open 
the socket to the Python server. It could be done more efficiently, but 
this was simpler (reusing some socket code I had lying around).
2. sendPacket sends the data in a packet, waits for the reply and puts 
the response into a field so I can check it is getting back the right 
answers. It also gives me the time between sending the packet and 
receiving the end of the result.

Results ?

First, and most importantly, it succeeded in the primary aim. I was able 
to make use of a library that exists and provides a lot of functionality 
that wouldn't otherwise have been available to me. And it allowed me to 
do so without writing C :-)

Second, it was reasonably easy to debug; each of the client and server 
could be run independently, with trace or debug output or using a 
debugger. The ability to track what was going on by monitoring the 
packets sent was very helpful.

Third, the efficiency was very good. The time for the whole operation 
(two  x  socket open + send + read) was around 42 msecs for the sample 
photo being used earlier (the one that took 1250 ms in Transcript). More 
importantly, a large photo took 480 msec versus 500 msecs for the naive 
C external.  Remember the imageData is NOT being passed over the 
interface;the filename is passed instead, and the Python code opens and 
reads the jpeg file. The jpeg file is around 1.5 Mb.
Note this photo is 3008 x 2008, so the imageData is 24 Mb. I tried 
passing the imageData and that added 250 msecs to the overall time - so 
even with that, it's still within 50% of the speed of a straightforward 
C implementation.

Fourth, it is very simple. Started yesterday evening, had it working by 
midnight, had it still working in about half the number of lines of code 
by 2 am :-)   Changing to a more efficient method (i.e. opening a single 
connection and sending multiple command / result pairs over it would) be 
a bit more work, and being less simple might be less robust initially.

Fifth, it offers lots of options. Develop Python functions and include 
them in a special-purpose server, to get more efficiency and easier 
coding; leave the server sparse and have all the flexibility. Do both. ....

Sixth, it's one more tool. I don't expect to use it often - but I have a 
couple of projects that I started in Rev and abandoned (i.e. abandoned 
use of Rev, switched to Python) which I could now do in 80% Rev + 20% 
Python with very little problem (both because of availability of a 
library in Python versus what was available in Rev).

Seventh (just because Mark W. mentioned it in the Q&A at RevCon), this 
uses a separate process, and so will allow Transcript stacks to take 
advantage of multiple processor machines (or multi-core CPUs), before 
Transcript is extended to provide threads.


I'd be happy to hear any comments on the approach, answer question, help 
anyone else who wants to play with it, etc. Or email the sample server 
and stack.

Finally - a short list of areas where I think Python offers a library 
that would be a good reason to try this approach. This isn't a list of 
why Python is "better" than Transcript.

1. Python Imaging Library
2. NumericArray (matrix math)
3. email library (i.e. rfc 822 headers, MIME encoding/decoding, etc.); 
also POP, IMAP and SMTP.
4. pysqlite - database access via SQLite
5. OpenGL
6. PDF generation (via ReportLab's toolkit)


Python server code:

> # This little program illustrates how easy it is to write network
> # servers using the classes in the SocketServer module.
> import SocketServer
>     
> def return_result_list(where, pList):
>     formatted = [ "%d" % x for x in pList]
>     where.write(",".join(formatted) + "\n")
>     
> class TranscriptHandler(SocketServer.StreamRequestHandler):
>     def handle(self):
>         global wfile
>         # Read lines of text, limiting each to 512 bytes.
>         # This will prevent someone trying to crash the server machine
>         # by sending megabytes of data.  
>         lines = []
>         while True:
>             new = self.rfile.readline(512).rstrip()
>             if new == "endofscript":
>                 break
>             lines.append(new)
>         text = "\n".join(lines)
>         wfile = self.wfile     # for use in user scripts, to 
> wfile.write() results.
>         exec(text)
>         self.wfile.write("endofresult\n")
>
> if __name__=='__main__':
>     # Create an instance of our server class
>     server=SocketServer.TCPServer( ('', 7779), TranscriptHandler)
>     # Enter an infinite loop, waiting for requests and then servicing 
> them.  
>     server.serve_forever()



-- 
Alex Tweedly       http://www.tweedly.net



-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.11/44 - Release Date: 08/07/2005




More information about the use-livecode mailing list