Algorithm time...

[Mike Bonner]

I have a curiosity question about this type of thing. Is there a way to
adjust things so that posting data to a url is not blocking? If so, you
could take advantage of a webservers multithreading (or several webservers)
to hand off "jobs" and then get the returned result.

Otherwise it should be possible using a server local to the machine to
write temp files to a directory, then "load" the url with a GET parameter
that points to the file in question with a callback message. Have an lc
server (or php or whatever) script handle each, return the data, at which
point your app can compile the results, sort, and be done with it. For
small datasets this might not be worthwhile, but if there is enough volume,
I think the returns would be worthwhile.


Ok, just tried this. With 500k lines of data, split into 10k line chunks
it's possible to do it in around 2.3 secs on my machine.  A hotter machine
with more cores is probably faster than that. Laying off really small
chunks doesn't pay, too many threads on the server side, 10k lines per
seems to be a good value.  15 to 20k per is very slightly faster. Of course
code would have to be tweaked to manage the very last group that might not
have the full number of 20k lines.

If you want to mess with something like this, the stack is here:
https://dl.dropbox.com/u/11957935/messingWithNumbers.livecode

And the lc server script is as follows:

<?lc
put 10 into tChunkSize -- 10 lines per chunk to process
put URL ("binfile:" & $_GET["tfile"]) into tData
if tData is not empty then -- make sure the file is not empty
set the itemdelimiter to tab
put the number of lines in tData into tCount -- number of lines sent.
 Should always be a factor of 10

-- process using the sort and grab method.
repeat tCount/tChunkSize times
put line 1 to tChunksize of tData into tTemp
delete line 1 to tChunksize of tData
sort tTemp numeric ascending by item 2 of each
put line 1 of tTemp & "<br>" & (the last line of tTemp) & "<br>" after
tNewData
end repeat
put tNewData
else
put "Problem finding data?"
end if
?>

Of course everything is set up to match my server set up.

Tried to just post the scripts but they ended up too large.  Twice.