Revolution and Task (handler) Overlap - Questions and special thanks to the latin expert and web police
gary.dennis at mantissa.com
Sat Feb 9 17:19:01 EST 2002
See comments, questions below.
----- Original Message -----
From: "Scott Raney" <raney at metacard.com>
To: <use-revolution at lists.runrev.com>
Sent: Friday, February 08, 2002 10:31 PM
Subject: Re: Revolution and Task (handler) Overlap
> On Fri, 8 Feb 2002 "Gary Dennis" <gary.dennis at mantissa.com> wrote:
> > Please forgive the minutia that follows but I am looking at developing
> > several high volume application in Revolution that will be based on the
> > examination of messages received over multiple TCP/IP connections.
> > I shoot myself (and the target application(s)) in the foot so to speak,
> > would like to get a few things straight in my mind.
> > The test case application receives console output from multiple IBM
> > mainframe systems and filters the console data to call operator
> > high priority messages. Because some of these mainframes have up to
> > active tasks the volume can be substantial (on the order of 5000
> > per minute). The average length of each message is approximately 300
> > bytes.
> Multiplying those together and you end up with just log files that are
> growing by 2GB *per day*. My first thought is that it might be worth
> investigating having those systems spew out a little less garbage ;-)
Thanks for the calculation (the reason for my original email). I didn't say
that the application would keep the data,only that it would receive and
screen. As for the data spew, the 800 tasks might be intellectual property
of 400 companies. It would be some task to contact all 400, tell them that
I had written an application using a tool that was incapable of managing
the combined message flow and asking them to make changes in the messages
their software produced to compensate for the less-than-stellar performance
of my application.
> > One thing I noticed early on is that I could not (even in a low volume
> > environment) let the message handler used to receive the console
> > actually process the message because of the speed with which the
> > could be received and stored. The environment is Win2k 512 ram, 1gHZ
> > processor speed.
> My second thought: don't use a Windows operating system for something
> that calls for a more robust solution. You'll get much better
> bandwidth through a Linux or BSD box (or any other UNIX box, for that
> matter) in this type of application.
Recall that the development tool is Revolution. Develop anywhere, run
anywhere; right? Thirty one years in software development tells me that you
develop on the least robust target platform, tune for the worst, and hope
that the customer can and will make the switch when the volume dictates.
If some of your following answers are true for all platforms, what real
difference would Linux make?
My goal is to make a decision on a cross-platform development tool for my
company not become an OS propeller head. I just watched 2 guys with 8 years
combined Unix/Linux experience mud wrestle with a portable, SUSE Linux, and
a $65 PCMCIA card for two days before getting it to connect to the LAN.
Life is too short, then you die.
> > At this point in the development, all the test case product actually did
> > a.) get the message buffer, b.) break the buffer down into lines (crlf
> > delimited message lines, the lf corresponds to the Rev RETURN, the cr
> > of the message is discarded) and c.) place the message lines AFTER a
> > scrollable field defined on a card. After a line limit is reached (set
> > 3000 lines), the handler would delete the FIRST line in the field.
> Another suggestion: if you're really dealing with a lot of rapidly
> changing data, don't even put all of it in fields. Just put the part
> you need displayed in there and keep the rest in a variable. Use a
> scrollbar object to allow the user to select what part is shown.
What am I missing? The field I'm using is scrollable. The idea was to give
the user a view anywhere inside the "retained" number of lines. Is there
some rev object that will give me live scrolling of variable content?
> > I
> > discovered that in I was using a replace for the mainframe cr
> > the entire buffer rather than the last character of each line. Causing
> > replace to reference the last character of each line rather than the
> > buffer made some difference but not enough.
> I don't follow this part, but the "replace" command is very fast and
> shouldn't be a bottleneck unless you're doing it unecessarily (e.g.,
> do it on a section *before* appending it, rather than appending and
> then doing the replace on the whole buffer again).
> > Question1. Is there a mechanism in Rev that causes all messages to be
> > stacked and executed in sequence as the targer handler exit is taken
> > the previous message? I had assumed that this is the case. Is there an
> > exception to this? Can you take a message off the message stack for a
> > handler and requeue it?
> Yes, no, and no.
So once you send a message, that message is handled next regardless of other
messages that may be received?
> > What I did was write a small message receive handler that passed the
> > message ( via "Send") to the formatting and storage handler. I am
> > that the exit of the handler will take place immediately after the READ
> > socket function and the handler exit is NOT dependent on the completion
> > the read. Question is.....
> This just seems like extra work to me. If the CPU has time to do it
> later (in addition to the extra overhead), why wouldn't it have even
> more time just to handle the data as it comes in? The only possible
> disadvantage I can see would be that you might get several "messages"
> in one read, but it seems to me this is likely to happen in any case
> so you've got to handle it anyway. Normal TCP buffering should take
> care of intermittent (lasting a second or two) overloads. If you
> really do need to do some "batch" processing on the data that takes a
> lot of CPU time, you should start thinking about offloading it to a
> different machine or at least to a different, lower priority, process
> if you must use only a single machine.
It wouldn't have more time later. This would be especially true if the
read is a BLOCKING read. If it is tell me now and I will give our copy of
REV to charity ( though that wouldn't be very charitable). 99 % of the time
we get more than one message. One operator command issued on such a
mainframe could generate 16 to 32K of message output. The advantage of the
send (if and only if the read is not a blocking read) could be that the
read wait time could be used by handler that was the target of the send. I'm
trying to figure out if REV is a popgun UI tool that I'm trying to use for
Rhino. Synchronous thinking in what has to be an async environment won't
really buy me anything.
> > Question 2. Does this approach give you any true overlap in the
> > receive-process cycle? It appeared to do so. My assumption is that the
> > engine is multithreaded in all cases with calls back to the engine
> > the opportunity for handler overlap.
> Sockets are no longer threaded as of engine version 2.4.1
> (subprocesses on Win32 are the only remaining threaded code in the
> engine, and that's only because the Win32 API is missing some key
> functionality in this area). We ripped all the threading stuff out
> because it was slow, significantly increased memory requirements, and
> was just less reliable in general. The apparent overlap is just
> because you're moving stuff from the front of the message queue to the
> end. But this operation isn't free, and just makes processing take
> longer overall. The more things you queue up with "send .. in", the
> worse the problem gets (keep in mind that it's got to sort through
> that queue each time through the main event loop).
No longer threaded for Win32 or for all target platforms? Slower for Win32
or all platforms?
Let me get this straight. You simply push the message to the end of the
queue so that...
read from socket x ... messagereceived
on messagereceived s,data
send " messageprocessor data"
read from socket s .... messagereceived
causes the sent message put to be pushed to the back of the queue until the
read is satisfied? Is the read truly a blocking read? If so, is it that
way for all platforms?
Slow is relative, significantly increased memory would be expected, and just
less reliable would usually relate to the quality of the code. Please tell
me that this was not done for all target platforms. We had planned to
recommend deployment under OSX. If this was done for all platforms, it
strikes me that you had some stellar goals that you gave up on.
> > Question 3. Is there some way to cause Rev to execute one handler at a
> > higher priority than other handlers?
Seriously? Not even the ability to specify that execution is ASYNC as it
relates to other handlers?
> > Question 4. Does the Number of lines function actually examine the
> > of the object to make the determination or is the value an attribute of
> > object?
> It's calculated, and so is not something you want to do a lot of.
> > In general are there Rev commands that cause high CPU utilization
> > when the amount of storage resident data reaches a high level?
> Not particularly, but as stated above, you want to avoid putting large
> amounts of data in fields, especially if the data is going to be
> changing often. Just keep it in variables for best performance.
> > Thanks......
> PS: I'd also recommend sending your WWW site designers out for
> "reeducation": I find it personally offensive that you can't even view
> www.mantissa.com except on a Windows system (it won't even open with
> browsers other than IE, and is all garbaged up on Macs even with IE).
> You'll not likely be getting any business from us, or any other users
> of UNIX or Mac systems, if you stick with this policy.
We design and write software, not web sites. Besides, if we had everything
(including web designers), where would we put it (or them)? If we did sell
our products to UNIX, Linux, or non-i.e. customers we would owe them a
refund since we have no such products. However, if those platforms are ever
supported, we will change the site. Until then, whenever you want to be
personally offended please visit our site.
> Scott Raney raney at metacard.com http://www.metacard.com
> MetaCard: You know, there's an easier way to do that...
> use-revolution mailing list
> use-revolution at lists.runrev.com
More information about the use-livecode