Revolution and Task (handler) Overlap

[Scott Raney]

On Fri, 8 Feb 2002 "Gary Dennis" <gary.dennis at mantissa.com> wrote:

> Please forgive the minutia that follows but I am looking at developing
> several high volume application in Revolution that will be based on the
> examination of messages received over  multiple TCP/IP connections.  Before
> I shoot myself  (and the target application(s)) in the foot so to speak, I
> would like to get a few things straight in my mind.
> 
> The test case application receives console output from multiple IBM
> mainframe systems and filters the console data to call operator attention to
> high priority messages.    Because some of these mainframes have up to 800
> active tasks the volume can be substantial  (on the order of 5000 messages
> per minute).  The average length of  each message is approximately 300
> bytes.

Multiplying those together and you end up with just log files that are
growing by 2GB *per day*.  My first thought is that it might be worth
investigating having those systems spew out a little less garbage ;-)

> One thing I noticed early on is that I could not (even in a  low volume test
> environment) let the message handler used to receive the console messages
> actually process the message because of the speed with which the messages
> could be received and stored.  The environment is Win2k 512 ram, 1gHZ
> processor speed.

My second thought: don't use a Windows operating system for something
that calls for a more robust solution.  You'll get much better
bandwidth through a Linux or BSD box (or any other UNIX box, for that
matter) in this type of application.

> At this point in the development, all the test case product actually did was
> a.) get the message buffer, b.) break the buffer down into lines (crlf
> delimited message lines, the lf corresponds to the Rev RETURN, the cr part
> of the message is discarded) and c.) place the  message lines AFTER a
> scrollable field defined on a card.  After a  line limit is reached (set at
> 3000 lines), the handler would delete the FIRST line in the field.

Another suggestion: if you're really dealing with a lot of rapidly
changing data, don't even put all of it in fields.  Just put the part
you need displayed in there and keep the rest in a variable.  Use a
scrollbar object to allow the user to select what part is shown.

>  I
> discovered that in  I was using a replace for the mainframe cr referencing
> the entire buffer rather than the last character of each line. Causing the
> replace to reference the last character of each line rather than the entire
> buffer made some difference but not enough.

I don't follow this part, but the "replace" command is very fast and
shouldn't be a bottleneck unless you're doing it unecessarily (e.g.,
do it on a section *before* appending it, rather than appending and
then doing the replace on the whole buffer again).

> Question1.  Is there a mechanism in Rev that causes all messages to be
> stacked and executed in sequence as the targer  handler exit is taken for
> the previous message?  I had assumed that this is the case.  Is there an
> exception to this?  Can you take a message off the message stack for a
> handler and requeue it?

Yes, no, and no.

> What I did was write a small message receive handler that  passed the
> message ( via "Send") to the formatting and storage  handler. I am assuming
> that the exit of the handler will take place immediately after the READ from
> socket function and the handler exit is NOT dependent on the completion of
> the read. Question is.....

This just seems like extra work to me.  If the CPU has time to do it
later (in addition to the extra overhead), why wouldn't it have even
more time just to handle the data as it comes in?  The only possible
disadvantage I can see would be that you might get several "messages"
in one read, but it seems to me this is likely to happen in any case
so you've got to handle it anyway.  Normal TCP buffering should take
care of intermittent (lasting a second or two) overloads.  If you
really do need to do some "batch" processing on the data that takes a
lot of CPU time, you should start thinking about offloading it to a
different machine or at least to a different, lower priority, process
if you must use only a single machine.

> Question 2.  Does this approach give you any true overlap in the
> receive-process cycle?  It appeared to do so.  My assumption is that the
> engine is multithreaded in all cases with calls back to the engine providing
> the opportunity for handler overlap.

Sockets are no longer threaded as of engine version 2.4.1
(subprocesses on Win32 are the only remaining threaded code in the
engine, and that's only because the Win32 API is missing some key
functionality in this area).  We ripped all the threading stuff out
because it was slow, significantly increased memory requirements, and
was just less reliable in general.  The apparent overlap is just
because you're moving stuff from the front of the message queue to the
end.  But this operation isn't free, and just makes processing take
longer overall.  The more things you queue up with "send .. in", the
worse the problem gets (keep in mind that it's got to sort through
that queue each time through the main event loop).

> Question 3.  Is there some way to cause Rev to execute one handler at a
> higher priority than other handlers?

No.

> Question 4. Does the Number of lines function actually examine the content
> of the object to make the determination or is the value an attribute of the
> object?

It's calculated, and so is not something you want to do a lot of.

>  In general are there Rev commands that  cause high CPU utilization
> when the amount of storage resident data reaches a high level?

Not particularly, but as stated above, you want to avoid putting large
amounts of data in fields, especially if the data is going to be
changing often.  Just keep it in variables for best performance.

> Thanks......

  Regards,
    Scott

PS: I'd also recommend sending your WWW site designers out for
"reeducation": I find it personally offensive that you can't even view
www.mantissa.com except on a Windows system (it won't even open with
browsers other than IE, and is all garbaged up on Macs even with IE).
You'll not likely be getting any business from us, or any other users
of UNIX or Mac systems, if you stick with this policy.

********************************************************
Scott Raney  raney at metacard.com  http://www.metacard.com
MetaCard: You know, there's an easier way to do that...