Wanted: Insights on implementing continuations. (Related to web frameworks)

[Andre Garzia]

Hi Folks,

sorry if this is a dupe, my address book is strange and I think the  
first mail I sent went to support at runrev.com instead of this list.  
Let me try again.

if anyone is reading this standing this standing please sit on your  
favorite chair because this will be a somewhat deep email that might  
need sometime to sink in and I'd value any comment or insight anyone  
can give me.

As many on this lists know, I've been building some web frameworks  
for sometime and I am reaching a point where I need some help. Most  
help I need now is on theory and good ideas. This email is touching  
the topics of continuations and why we need them for webframeworks.  
Continuations are not a familiar concept for xTalk coders, actually I  
think that most people familiar with that will be from coding in  
functional languages such as Scheme.

I am no expert in continuations, not even close, I am newbie on that  
as I can be.

Imagine that you have this piece of code:

ask "what is your name?"

This piece of code would use the ask dialog box, pick the result and  
return the result inside the it variable to the handler that called  
it. Pretty simple standard behaviour, right? Well now imagine that  
you want to call a function but you want that the result of that  
function should be routed to some other place. So you name your  
function and the result should be routed to the CONTINUATION. I am  
sure this is not the best way to talk about such concept, this not  
exactly the full concept. The right concept might be thinking about  
Futures. For a simple address book database we might be asking the  
user many questions like what is his name, age, email and stuff. The  
idea about continuations is that when you ask it's name, it knows  
that in the future it will ask it's age and so on. I really wished I  
had this concept better understood so that I could talk it better.

One way to think about what I am here styling as continuations is to  
think about flows, the work flow, it goes from present to future. The  
way the web apps usually works is going from past to present. Most  
web apps will keep track of what's been done because they don't have  
a continuation mechanism. So everytime your web app launches and this  
happens a lot with CGI it needs to poke it's cookies, session vars  
and stuff to discover what's been done and where to go from that point.

If you think of the web using Revolution metaphors like each page is  
a card. And you have a card with a quiz and some other card with  
additional info that you'd be showing depending on the user response.  
One way to work would be showing the first form, driving the answers  
to a cgi that would save them to a session then depending on the  
variables (poking into the past) it would render the additional form  
and pass that to a third CGI that would get the new additional form  
data and restore the former session from the previous cgi thus  
summing everything into a variable soup that it could use. This is  
how 90% of the web guys are doing. I want it more elegant.

The main issue with the web is that we cannot push data into the  
browser after it closes de connection, we can hack our way using ajax  
techniques but yet, we would not be pushing data, ajax would be  
retrieving it. Why am I so concerned about this? think of a  
colopicker, in a web form you might have a field that is a color  
picker, when you click it it refreshes the page to show you a very  
nice color picker and then it picks your chosen color and return to  
the same form as before with your color value there. If we had  
support for continuations and session saving restoring, we could  
create that. As it is, we would be doing a lot of session saving,  
poking into the past, checking things and just to have our web app  
broken as soon as some user hit the back button (thus destroying the  
session.)

What I want to do is to make the following pseudo-code possible:

on fiddleWithColors
	WebAskColor "choose color for background?"
	set the backgroundcolor to it
	WebDisplayWithMessage "new background color."
end fiddleWithColors

What is the problem with this thing, well, there are two trips to the  
browser, one WebAskColor would render a color picker and would pick  
the answer and return to the same handler. This is currently  
impossible because after rendering the HTTP connection is broken, the  
guy picks the color, a new request is created and sent to the server.  
No way you could talk to the browser and pick answers in the same  
handler. The way we currently work in libCGI and libFCGI is by  
relying on callbacks, we send something and tags the handler that  
will be called afterward which is the continuation for that handler.  
What we would do is to have a handler to render the color picker  
form, a cgi logic to pick the answer and change the display for the  
new form, we would need at least two handlers in libCGI. You could  
ask me: "What is wrong with having a function and a continuation  
callback?", well the answer is, what happens if you're implementing  
some big project with a complex workflow? Do you want to have a  
callback for each thing, for each step a render and a callback?  
Imagine a wizard like web app, that goes gathering data theu a range  
of 38 pages and 200 questions, like a class exam... one callback per  
page, saving state forward and deciding what to show... Thats why we  
must think of this problem if we ever want our Revolution web work to  
scale to enterprise.

If we had multi thread or forking support, I could spawn a new  
proccess/thread for each request and use debugging stepper like  
techniques to go thru the code rendering things as necessary and  
stepping thru the code as needed. Without multi thread or forking  
support, I can't afford to lock the engine inside a handler waiting  
for the browser to return data, this would clog all the other users.

So we're locked in a monothread-no-process-for-you engine and we know  
that coding using callbacks for every single action our software  
needs to take is not the best way around, how should we work? Should  
we implement a pre-processor that would pick code like the above and  
re-factor it using callbacks, much easier to write? Should we build  
some kind of save state machine able to save globals and vars and  
stack state and keep saving/restoring at each cycle?

Coding in xTalk is simple, coding web apps in xTalk must also be  
simple or we'll be loosing something. The key to keep xTalk simple is  
not to make the developer waste it's time with coding things that are  
not related to the problem he is trying to solve. That is what moved  
me away from C/C++, I was so overhelmed with memory managemend and  
keep tracking of which pointer is pointing at who that I was not  
actually doing anything related to solving the problem I had at hand.  
If when the xTalk dev is trying to fiddle with web apps and he needs  
to manually keep track of sessions and generate his own functions to  
poke the system so that his app can discover where in the work flow  
it is, then he'll be spending 70% of his time doing this things and  
30% of his time actually doing his app. If we had a smart way to do  
continuations, to tag, you go from here, to there and your state is  
safe even if the user clicks that silly back button, then we would be  
actually ahead of many technologies out there...

did anyone understood what I am talking about? it's 4:11 AM here in  
brazil and I haven't slept yet. Spent the whole night reading about  
RoR, PHP, JSP, Servlets, Seaside, Call/CC and all those papers got me  
thinking too much, I decided to ask for comments before settling on  
any technique...

Cheers
andre