[revServer] process timeout issue

[Jim Ault]

The one provision ( or gotcha ) that I would add to this discussion is
         log files

You should insure that log files to not become too large, especially  
if they are using an XML format that is very fluffy.

I just had a client decide to program his ping schedule into my system  
for once every 3 seconds instead of the design of 50 per hour. One of  
my log files expanded to 42 Mb and caused a connection timeout  
( greater than 10 seconds ).  I now have taken protective action to  
protect my dedicated server, but the shared hosting part of the  
network cannot be changed ( the admin company is not willing to do  
this )

Most log file logic is to add new transactions to the end of an  
existing file, thus 2 Mb requires more time than 100K.  Some servers  
start new log files every calendar day, but others may have a much  
lower frequency.

You may not be able to control this frequency, but you might be able  
to switch modes.
Sometimes all events are logged and designated 'verbose' or 'debugging  
mode'.

If you are on a shared host, you may not have the permissions to edit  
or delete these files.  On your own server, you should investigate if  
the log files could expand quickly, especially if you are doing a week  
of intense testing and programming.





On Aug 6, 2010, at 9:55 AM, Mark Talluto wrote:

> On Aug 4, 2010, at 7:53 AM, Richard Gaskin wrote:
>
>> I've been experimenting with spidering, data mining, and analytics,  
>> and like any processor-intensive tasks it would never occur to me  
>> to put them on a shared host.
>>
>> Like many hosts, the one I'm using offers dedicated servers for  
>> less than $70/mo, but being a cheapskate I've gone one step further  
>> during this experimental phase:  I bought a nettop off Ebay for  
>> just $150, set it up with Ubuntu and Rev, and that does all the  
>> heavy lifting 24/7, posting only the output from those process to  
>> my servers periodically as needed.
>>
>> I never run into the CPU cycle limits most hosts have on their  
>> servers, and I don't even slow down my own web server from its  
>> tasks of serving pages to my visitors and handling their purchases.
>>
>> When the workflow expands to required tighter integration between  
>> the processing and the output, I can move the system from my office  
>> to a dedicated server with multiple redundant fat-pipe connections  
>> for just a few bucks a month.
>>
>> There are a million ways to create robust scalable infrastructures  
>> to handle any load.  Many are cheap and easy to do, and for most of  
>> those tasks you can do them all in one fun language.
>
> We have been using this technique for years.  We even posted the  
> application we use to do this task in RevNet.  I believe I need to  
> update that file now that I think of it.  But in short, we use our  
> ISP to gather orders.  Our client software sends a request for a  
> key.  Our local computer in the lab just pings the directory on the  
> ISP every 4 seconds and downloads all the orders in that given  
> directory.  The heavy lifting and database work is done on a  
> computer in the lab.  The key is then sent back up to the ISP where  
> the client computer is checking in for the result of that work every  
> 4 seconds.  The whole thing works out nicely and we keep our CPU  
> usage low.
>
> Mark Talluto

Jim Ault
Las Vegas