Fw: application slowing down

[Dar Scott]

On Tuesday, August 20, 2002, at 11:19 AM, Rich Mooney wrote:

> I'm desperate and throwing myself on the mercy of this mail list.  
> I have an
>> irate customer who wants this problem solved quickly.  I have used
> MetaCard
>> to create software which we use to monitor and control our distributed
>> lighting control systems.  It does these things very well.  I 
>> currently
> have
>> a job which requires that I poll the status of all my controls 
>> every 30
>> seconds.  I do this so I can catch any changes of state and a close
>> approximation of the time at which that change occurred.  I also 
>> use this
>> function to catch any errors in operation.  The problem is that 
>> the time
> it
>> takes to poll all the controls becomes increasingly longer the 
>> longer I
> run
>> the software.  Initially it will take 33 seconds to poll all of my
> controls
>> if the polling interval is set to 30 seconds.  After three hours 
>> it will
>> take 43 seconds.  By the next day it will take 2 minutes.

Is your network TCP/IP?  Besides the processor usage you can 
monitor network usage and this may give a clue.  A packet sniffer 
might also.

Is each poll run by a batch of "send in <time>"?  If it takes 33 
seconds and a poll is started every 30 seconds, does that mean 
there is an overlap?  That can cause a few problems.

First of all each poll might take a tiny bit longer than the 
previous one if it competes for processor resources.  This is 
because the previous one overlaps.  This will cause that poll to 
overlap the next a little more, because it has been slowed down 
some.  Perhaps that is the growth you see.

Second, unless you carry all of your poll state in args of the 
handler spec'd in send (or other wise provide for this problem), 
you might have one send cycle messing up data from the next.

Third, if the dialog is command-response, there might be a risk of 
getting a response that is from a query of another send cycle.  
Handling those might cause more delays and use up a tiny bit more 
processor resource.

If this looks like it could be your problem, you might want to 
prevent a send cycle from starting if one is in progress or start 
them every, say, 45 seconds.  For now.

Then look for ways to get polling down under 30 seconds.  For 
example, you might query several controls at once.  You might want 
to trim the fat from your code.  You might enhance some hardware.

In the long term, you might want to put polling on leaf nodes only 
and distribute these over the system.  They can communicate with 
concentrators and central control in better ways than polling.  
Polling has killed many large system designs.

I made a lot of wild guesses.  (For example, I assumed that polling 
is started every 30 seconds, not that there is a 30 second delay 
from the end of one to the start of the next.)  Since you have a 
customer on your back, I figured that is the way to go.  I hope I 
got close to the mark.

Dar Scott