having to help Rev (was: Re: Memory Leak on export png????)

[Richard Gaskin]

Dave persisted:
> On 22 Mar 2007, at 18:29, Richard Gaskin wrote:
>> In the ten years I've been working with this engine this is the  
>> first verified leak I've seen.  Let's be generous and say that  
>> maybe one or two others might have been discovered in that time.   
>> Even then, over a decade that's really quite good -- and  
>> accomplished without automated stress testing.
> 
> There were three problems, the leak was just one of them.

The third "problem" (garbage collection appearing to only be done at 
idle) was based on a misunderstanding of results and turned out to have 
no supporting evidence in this case, as I noted earlier:
<http://lists.runrev.com/pipermail/use-revolution/2007-March/095651.html>

That leaves only two, both of which require multiple iterations to be 
evident.

>> The export command appears to work well when run once or even a  
>> dozen times.  Unit testing should always be done, and in this case  
>> would yield a good result.  Only a sustained test with a great many  
>> iterations will expose this specific problem, and only in the Rev  
>> IDE.  The leak doesn't exist in a standalone or in other IDEs, and  
>> since some issues may be specific to standalones it would be  
>> necessary to run any soak tests in at least those two environments.
> 
> Not really if you were to write files 1 to 300 you would hit it at  
> 288 and I had it happen earlier than that to start with. 

Agreed:  more than 288 iterations would be needed to see that problem.

> In fact the memory leak would be visible straight away, all you
> have to do is run once it and look at the memory allocations.

That memory fluctuates while a program is running is normal.  The 
cumulative effect in which some of that memory isn't released is only 
evident with multiple iterations.

>> Consider all the effort to design, build, and run these tests,  
>> release after release, year after year, and after a decade of that  
>> we might turn up a leak or two and maybe a relative handful of  
>> other errors of the sort which can be exposed through that specific  
>> form of testing.
> 
> Took me 10 minutes to build the test for the export snapshot command  
> and 2 minutes to run it. On the first part I was working on (last  
> week) it took me 30 minutes to build the tests and about the same to  
> run the tests. I then ran it at least once a day (after I'd added/ 
> changed things) to make sure I hadn't broken something.

Hindsight.  I'm sure you're aware that good soak tests commonly run far 
longer than 2 minutes, and in most cases for good reason.

That this one isolated case was discoverable in less is as much of an 
anomaly as the rarity of the bug itself.

>>> Another problem here is that people may have different ideas on  
>>> what  "Beta" means and I haven't seen it defined in terms of  
>>> RunRev. One  company I worked for defined it as meaning "Feature  
>>> Complete, No  Known Crashing Bugs".
>>
>> That's the ideal, but I've known no company that ships every Beta  
>> in a state that meets that definition.
> 
> Well, I've beta tested Photoshop and I AFAIK there were no known  
> crashing bugs and AFAIR it was feature complete.

You wrote "Known" crashing bugs. That would require a level of testing 
rarely if ever possible in commercial application development.  In fact 
other apps from Adobe (and other large companies as well) have been 
delivered to Beta with bugs which were discovered to cause crashes, and 
crashing issues sometimes even survive undetected into final builds.

>> I've participated in tests for Adobe, Microsoft, Apple, Oracle, and  
>> others who have shipped Betas while still developing new features.
> 
> "Feature Complete" was just the way that company did it, I've also  
> seen that beta versions that still being developed. I was trying to  
> find out what "beta" meant in the wonderful world of RunRev.

It seems their definition is in keeping with industry norms.

>> That you're able to run stress tests on all features and identify  
>> 100% of requirements to complete satisfaction before your first  
>> Beta is quite an accomplishment,
> 
> Could you tell me where I said that?

The sum of your posts suggest an expectation for other developers of 
that level of effort.  It would seem reasonable that such expectations 
are at least met in your own shop.

> I run Stress Tests while developing my software

Great.  Another 10,000 function points and your work will begin to 
approach the complexity of Rev.

> However stress testing is not an accomplishment at all it's really  
> easy

...for small programs, or with selective testing possible only with 
hindsight.

> that's why I really can't see why you are going on about it

You might review your posts to see where this ongoing discussion of 
stress testing originated.  You're welcome to stop going on about it at 
any time.

-- 
  Richard Gaskin
  Fourth World Media Corporation
  ___________________________________________________________
  Ambassador at FourthWorld.com       http://www.FourthWorld.com