Remotes - the new, exciting alternative to Externals.
Alex Tweedly
alex at tweedly.net
Sat Jul 9 17:28:42 EDT 2005
I've been playing with writing externals for Revolution lately. It's
been interesting. I'm very glad to know I can do it, and should I ever
need to, I will. But I can't say it's been a lot of fun. Writing C code
is frustrating and not very productive in the best of circumstances
(e.g. supported by a symbolic debugger, protected processes, etc.) In
the circumstance of running an external with Rev's IDE (where many
errors cause immediate termination of the IDE, with no diagnostic info
provided at all, far less any chance to debug), it is very unproductive.
So while it's good to know that externals are possible for me, I also
know that they are to be avoided if reasonable alternatives exist.
So I started looking for alternatives .....
Reasons to write externals in the first place.
1. to get access to hardware or other facilities not otherwise available
2. to get higher performance (e.g. matrix ops, image processing, etc.)
3. to take advantage of existing library packages
4. other significant but less common reasons
As described above, I don't like C (far less C++). The language I do
like (assuming Transcript isn't an option) is Python.
Python is, in many ways, similar to Transcript. They're both scripting
language, dynamic types (or typeless), associative arrays, etc. - that's
largely why I like them both. But they're also quite different, and
complementary, in many ways. So, without falling into the trap of
becoming a comparison between the two languages, I believe that a number
of the reasons for writing an external would equally apply to using Python.
Of course, it would be difficult to write an actual external in Python.
But I realized that one of the strengths of both Python and Transcript
is that they have good, simple socket facilities. So I decided to try
"Remotes" - socket-based Python add-ons to Transcript.
I wrote a small (30 lines) Python server which accepts "commands" over a
TCP socket, and returns its result over the same TCP connection (see
the end of the email for the source code of the Python program). The
"commands" are in fact snippets of Python code; so I can develop the
Transcript stack and just change the snippets of Python to be sent and
executed, without even restarting the server. So the whole process is as
dynamic as Rev usually is.
As an example, I used the Python Imaging Library to calculate the
Histogram of a JPEG, done as two steps (to avoid making it too easy). So
there are two buttons - one to open and specify a file, the other to
calculate th histogram for it.
The script of the first button is
> on mouseUp
> local t
> answer file empty with filter "JPEGs,*.jpg"
> put it into theFile
> put "import PIL.Image" & cr \
> & "from PIL import *" & cr \
> & "global im" & cr \
> & "im = Image.open(" & quote & theFile & quote & ")" & cr into
> theScript
> put theScript & cr & "endofscript" & cr into t
> reStart
>
> SendPacket t
> end mouseUp
While the second is
> on mouseUp
> local t
> reStart
> put "hi = im.histogram()" & cr \
> & "return_result_list(wfile, hi)" into theScript
> put theScript & cr & "endofscript" & cr into t
> SendPacket t
> end mouseUp
Notes.
1. The "restart" handler is used to reset any open sockets, then open
the socket to the Python server. It could be done more efficiently, but
this was simpler (reusing some socket code I had lying around).
2. sendPacket sends the data in a packet, waits for the reply and puts
the response into a field so I can check it is getting back the right
answers. It also gives me the time between sending the packet and
receiving the end of the result.
Results ?
First, and most importantly, it succeeded in the primary aim. I was able
to make use of a library that exists and provides a lot of functionality
that wouldn't otherwise have been available to me. And it allowed me to
do so without writing C :-)
Second, it was reasonably easy to debug; each of the client and server
could be run independently, with trace or debug output or using a
debugger. The ability to track what was going on by monitoring the
packets sent was very helpful.
Third, the efficiency was very good. The time for the whole operation
(two x socket open + send + read) was around 42 msecs for the sample
photo being used earlier (the one that took 1250 ms in Transcript). More
importantly, a large photo took 480 msec versus 500 msecs for the naive
C external. Remember the imageData is NOT being passed over the
interface;the filename is passed instead, and the Python code opens and
reads the jpeg file. The jpeg file is around 1.5 Mb.
Note this photo is 3008 x 2008, so the imageData is 24 Mb. I tried
passing the imageData and that added 250 msecs to the overall time - so
even with that, it's still within 50% of the speed of a straightforward
C implementation.
Fourth, it is very simple. Started yesterday evening, had it working by
midnight, had it still working in about half the number of lines of code
by 2 am :-) Changing to a more efficient method (i.e. opening a single
connection and sending multiple command / result pairs over it would) be
a bit more work, and being less simple might be less robust initially.
Fifth, it offers lots of options. Develop Python functions and include
them in a special-purpose server, to get more efficiency and easier
coding; leave the server sparse and have all the flexibility. Do both. ....
Sixth, it's one more tool. I don't expect to use it often - but I have a
couple of projects that I started in Rev and abandoned (i.e. abandoned
use of Rev, switched to Python) which I could now do in 80% Rev + 20%
Python with very little problem (both because of availability of a
library in Python versus what was available in Rev).
Seventh (just because Mark W. mentioned it in the Q&A at RevCon), this
uses a separate process, and so will allow Transcript stacks to take
advantage of multiple processor machines (or multi-core CPUs), before
Transcript is extended to provide threads.
I'd be happy to hear any comments on the approach, answer question, help
anyone else who wants to play with it, etc. Or email the sample server
and stack.
Finally - a short list of areas where I think Python offers a library
that would be a good reason to try this approach. This isn't a list of
why Python is "better" than Transcript.
1. Python Imaging Library
2. NumericArray (matrix math)
3. email library (i.e. rfc 822 headers, MIME encoding/decoding, etc.);
also POP, IMAP and SMTP.
4. pysqlite - database access via SQLite
5. OpenGL
6. PDF generation (via ReportLab's toolkit)
Python server code:
> # This little program illustrates how easy it is to write network
> # servers using the classes in the SocketServer module.
> import SocketServer
>
> def return_result_list(where, pList):
> formatted = [ "%d" % x for x in pList]
> where.write(",".join(formatted) + "\n")
>
> class TranscriptHandler(SocketServer.StreamRequestHandler):
> def handle(self):
> global wfile
> # Read lines of text, limiting each to 512 bytes.
> # This will prevent someone trying to crash the server machine
> # by sending megabytes of data.
> lines = []
> while True:
> new = self.rfile.readline(512).rstrip()
> if new == "endofscript":
> break
> lines.append(new)
> text = "\n".join(lines)
> wfile = self.wfile # for use in user scripts, to
> wfile.write() results.
> exec(text)
> self.wfile.write("endofresult\n")
>
> if __name__=='__main__':
> # Create an instance of our server class
> server=SocketServer.TCPServer( ('', 7779), TranscriptHandler)
> # Enter an infinite loop, waiting for requests and then servicing
> them.
> server.serve_forever()
--
Alex Tweedly http://www.tweedly.net
--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.11/44 - Release Date: 08/07/2005
More information about the use-livecode
mailing list