Chrome, lchttpd and a possible Livecode socket bug?

[Richard Gaskin]

David Bovill wrote:

 > @andre - dropbox is no longer public. I'd love to check out your old
 > server

Andre's been immersed in the Firefox project, and doesn't frequent this 
list as he used to.  Back when he did he wrote at least two lengthy 
posts here about why he stopped development of his httpd stack, and why 
he doesn't recommend using it for Web servers.

In short, serving browser clients is especially challenging in a 
non-threaded and non-forking engine like LiveCode, because what we think 
of as a single transaction is often a dozen or more - in addition to the 
HTML page it'll often include links to CSS, JavaScript, and any number 
of images files, resulting in many requests for a single page.

Because the engine is single-threaded and non-forking, all requests are 
effectively queued.  Network I/O is somewhat non-blocking, but 
everything else the engine does, such as file I/O, is blocking, so 
handling things like Web pages with their multiple requests becomes more 
onerous than just using any of the apps already doing a good job for 
that, like Apache, NGineX, etc.

Using LC as a CGI under Apache is a reasonably good use case, as it 
allows concurrency under Apache's forking.  But replacing Apache with LC 
is perhaps more trouble than it's worth, at least until we can do 
forking ourselves.


That said, I do believe there's good value in exploring LiveCode daemons 
for other uses beyond Web servers.

Application-specific servers, where transactions will be a single 
request rather than a dozen or more as is common with Web pages, can be 
very good fit for LiveCode.  Even better when the client is also made 
with LC, so we can take advantage of things like compressed encoded 
arrays for much more efficient throughput than translating to/from JSON.

In that sense, whether we use port 80 with HTTP conventions or make a 
protocol of our own over another port, we're making a simple socket 
server, and in my tests LC does well with that sort of thing.

One of my favorite examples along these lines is EVE Online, since it's 
one of the world's most popular MMOGs and is served by a single Python 
engine instance.

In all fairness, though, since Python is like LiveCode in that it 
doesn't support true concurrency, a special build of Python was needed, 
CPython, which includes support for "green threads" that provide 
concurrency but with less overhead than system-managed threads.

A similar enhancement could be made with LiveCode, and I hope it comes 
about some day since it would allow us to do everything Python can do.

But in the here-and-now, few of us need to solve a C10k problem, and 
with reasonable traffic a single LiveCode daemon can do well.

If traffic grows beyond what a single instance can handle, the 
bottlenecks can be distributed among any number of other LC instances 
managed by the daemon via local sockets, as we've discussed here before.


To explore the potential for building socket servers with LC, I've found 
two projects especially helpful: mchttp and the chat example.

mchttp
------
Both Andre and Todd started their httpd projects as forks of Scott 
Raney's mchttpd.  Dr. Raney's original stack no longer works because 
most browsers today have stricter requirements for header info than his 
  script provided, but with his permission I added only Andre's header 
fix to Raney's original script and the result is available under MIT 
license here:
<http://fourthworld.net/lc/mchttpd-4W.zip>

Andre's and Todd's enhancements are likely very useful; I'm providing 
the link to Raney's fixed original only because if there seems to be an 
issue in how the engine is handling sockets this simpler script may make 
diagnosis easier.

chat
----
The chat client and server code in this lesson is about as simple as it 
gets, making it a good starting point for expansion since it was part of 
a client deliverable and is known to work well:
<http://lessons.runrev.com/m/4071/l/12924-how-to-communicate-with-other-applications-using-sockets>

While the mchttp stack is a good example of simple transaction handling 
in which a single client makes a request that is returned only to that 
client, the chat example illustrates broadcasting in which any data 
coming in from any client is broadcast to all of them.

Between these two models we have a nice foundation for a great many 
possible types of interesting and useful client-server applications.


With the socket issue you've seen, it may be useful to apply the logging 
you have to either mchttpd or the chat example and see what you find.

I've spent very little time with mchttpd, but I did some stress testing 
on the chat server a while back and it held up admirably: I had three 
separate client computers throwing as many requests as they could to the 
daemon running on one of my VPSes for two minutes, monitoring progress 
in htop as it ran.  The daemon never used more than 25% of the CPU, and 
when the barrage ended it reclaimed all memory and dropped back to zero CPU.

Hopefully you'll find a quick fix in one of the scripts, and if it does 
indeed turn out to be an engine issue your bug report to the RQCC is 
much appreciated.

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com