"fork" command?

[Mark Waddingham]

On 2016-01-07 20:03, Richard Gaskin wrote:
> I'm just far enough into Robert Love's "Linux System Programming" that
> I think the solution to FastCGI may be much simpler than I'd
> previously thought.

I think you need to read a bit more about fork ;)

> I think we need a new command that launches a specified process but in
> a way that uses a call to "fork" to pass file descriptors (which
> include sockets and other I/O info) to the child process.

The 'fork()' system call is a very low-level primitive which is the 
basis of executing other processes on UNIX based systems. You have to 
use it very carefully - indeed, pretty much any use of fork will almost 
immediately be followed by a call to some variant of 'exec', which 
basically runs a different executable in the current process, replacing 
the original one.

> In many ways it would work very similarly to the existing "open
> process", but allow params to give the child process access to things
> like socket connections, pipes, files, etc. the parent process has
> access to at the time the child process is launched.

Well 'gives access to' is slightly misleading here as it suggests that 
you can use them all, which isn't really true. When you fork the kernel:

1) clones the task structure (the thing the kernel uses to represent a 
process)

2) marks all memory pages as copy on write

3) increases the reference count on every thing attached to a file 
descriptor

This means that when the child process starts running its memory image 
is identical to the parent, and it has *exactly the same* file 
descriptors as the parent.

Critically, any state which the original process has related to 
connections to things (e.g. databases or display servers) is the same in 
parent and child, which means they have the same 'state' on the other 
end of the connection. This means that any usage of them in either after 
the fork will cause things to break, probably horrendously as the two 
processes run asynchronously.

This means that you need to engineer things so that the child gets 
appropriate fd's and the parent keeps appropriate fd's. The typical 
thing you do here is faff around with pipes. For example, here is the 
critical bit of code from the engine's open process command:

         int tochild[2];
         int toparent[2];
         int4 index = MCnprocesses;
         if (pipe(tochild) == 0)
         {
             if (pipe(toparent) == 0)
             {
                 MCU_realloc((char **)&MCprocesses, MCnprocesses,
                             MCnprocesses + 1, sizeof(Streamnode));
                 MCprocesses[MCnprocesses].name = strclone("shell");
                 MCprocesses[MCnprocesses].mode = OM_NEITHER;
                 MCprocesses[MCnprocesses].ohandle = NULL;
                 MCprocesses[MCnprocesses].ihandle = NULL;
                 if ((MCprocesses[MCnprocesses++].pid = fork()) == 0)
                 {
                     close(tochild[1]);
                     close(0);
                     dup(tochild[0]);
                     close(tochild[0]);
                     close(toparent[0]);
                     close(1);
                     dup(toparent[1]);
                     close(2);
                     dup(toparent[1]);
                     close(toparent[1]);
                     execl(MCshellcmd, MCshellcmd, "-s", NULL);
                     _exit(-1);
                 }
                 MCS_checkprocesses();
                 close(tochild[0]);
                 char *str = path2utf(ep.getsvalue().clone());
                 write(tochild[1], str, strlen(str));
                 delete str;
                 write(tochild[1], "\n", 1);
                 close(tochild[1]);
                 close(toparent[1]);
                 MCS_nodelay(toparent[0]);
                 if (MCprocesses[index].pid == -1)
                 {
                     if (MCprocesses[index].pid > 0)
                         MCS_kill(MCprocesses[index].pid, SIGKILL);
                     MCprocesses[index].pid = 0;
                     MCeerror->add(EE_SHELL_BADCOMMAND, 0, 0, 
ep.getsvalue());
                     return IO_ERROR;
                 }
             }
             else
             {
                 close(tochild[0]);
                 close(tochild[1]);
                 MCeerror->add(EE_SHELL_BADCOMMAND, 0, 0, 
ep.getsvalue());
                 return IO_ERROR;
             }
         }


> But for those of you more familiar with Linux system programming, do I
> misunderstand the difficulty involved?

Yes - fork() isn't what you are looking for. It isn't magical - it does 
*precisely* what it says it does which is insufficient for what you are 
proposing (and isn't what it is really used for anywhere).

> Forking seems so common in other tools, and not having it appears to
> be the one detail standing between where we are now and having not
> just FastCGI, but also being able to build truly excellent application
> servers on par with Node.js and other similar systems.

I'd say forking (except the purpose of implementing the equivalent of 
shell or open process) is actually very rare. Any language which offers 
bindings to system libraries and such will likely have os.fork() or some 
such for completeness - however this doesn't mean it is actually used 
very much (nor should it - there are few patterns around fork which are 
safe and generally useful - shell and open process pretty much cover 99% 
of them).

> LiveCode is a great language, and if we had the ability to fork we
> should be able to build a wide range of powerful, scalable, efficient
> systems, breaking far beyond the limitations of CGI we're limited to
> now.

Using LiveCode in the contexts you are talking about doesn't require 
fork().

LiveCode can almost be used as a Node.js type server already - start a 
-ui standalone engine running which 'accepts connections' and dispatches 
requests for appropriate protocols out to whatever you need (the missing 
bit, as I've mentioned before is that some 'long' running things such as 
db access block, rather than do things asynchronously). Like other 
Node.js style setups, I'd imagine you'd need some sort of 
'load-balancing' magic to farm the Node.js-like processes which farm the 
requests - I'm sure others know more about this than me.

Adding FastCGI support to LiveCode server would also be a similar 
solution (where the blocking nature of some aspects of LiveCode wouldn't 
matter so much). When you run this through something such as mod_fcgi, 
you get the 'farming' of processes for free, so in LiveCode you only 
have to worry about servicing the requests, rather than how processes 
get started up and shutdown and used.

Basically, managing collections of processes to fulfil CGI-like requests 
(which is what fork() - because it is what you use to do open process - 
is used for in these contexts) is already a solved problem - there is no 
need for it to be solved again.

> If all we need is a new command to wrap the Linux "fork" call, after I
> finish Love's book I may brush up on my C skills and give it a go.

No need to 'brush up' on your C skills. Something along the lines of the 
following in LiveCode Builder should work:

library module org.livecode.fork

foreign handler _Fork() returns CInt binds to "fork"

public handle Fork()
   return _Fork()
end handler

end module

Although, as mentioned above, it won't get you very far for the reasons 
outlined above.

Warmest Regards,

Mark.

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps