having to help Rev (was: Re: Memory Leak on export png????)

[Dave]

On 22 Mar 2007, at 18:29, Richard Gaskin wrote:

> Dave continued:
>
>> I suppose my point is that in order to have a beta test, you need  
>> to  have gone thru the steps to get there. It's no good producing   
>> mountains of code, doing little or no testing at the development   
>> phase and then throwing that out for beta.
>
> With 20/20 hindsight it's easy to suggest that stress testing would  
> have found this leak, and indeed it might well have.  But I don't  
> think it would be a cost-effective practice for RunRev to adopt  
> across this product.
>
> In the ten years I've been working with this engine this is the  
> first verified leak I've seen.  Let's be generous and say that  
> maybe one or two others might have been discovered in that time.   
> Even then, over a decade that's really quite good -- and  
> accomplished without automated stress testing.

There were three problems, the leak was just one of them.

> The export command appears to work well when run once or even a  
> dozen times.  Unit testing should always be done, and in this case  
> would yield a good result.  Only a sustained test with a great many  
> iterations will expose this specific problem, and only in the Rev  
> IDE.  The leak doesn't exist in a standalone or in other IDEs, and  
> since some issues may be specific to standalones it would be  
> necessary to run any soak tests in at least those two environments.

Not really if you were to write files 1 to 300 you would hit it at  
288 and I had it happen earlier than that to start with. In fact the  
memory leak would be visible straight away, all you have to do is run  
once it and look at the memory allocations.

> And because Rev supports multiple operating systems, each with  
> differing APIs and idiosyncrasies, each of those two tests would  
> need to be run on all supported OSes.  Here we already have a  
> combinatorial explosion of test scenarios:  2 engines X Win 98,  
> Win2K, WinME, WinXP, Vista, Mac OS 9, OS X 10.2, OS X 10.3, OS X  
> 10.4, at least one Linux, and for anything that involves QT  
> multiply 9 of those by the number of supported QT versions.  That's  
> 20 tests without QT, and just for one command.

Agreed. To test it all those platforms would be hard. But in this  
case it just needed to be tested on the development machine of the  
person doing the coding and stressed for a lot of  operations. I do  
this automatically without even thinking about it.

I agree that you'd have to run it under the IDE and as a Standalone.  
To start with you just run it in the IDE, then at some point when you  
feel the time is right you build a standalone. To build a standalone  
for Mac and Windows takes literally seconds and in the case of the  
export snapshot the tests take 2 minutes to run.
>
> Consider all the effort to design, build, and run these tests,  
> release after release, year after year, and after a decade of that  
> we might turn up a leak or two and maybe a relative handful of  
> other errors of the sort which can be exposed through that specific  
> form of testing.

Took me 10 minutes to build the test for the export snapshot command  
and 2 minutes to run it. On the first part I was working on (last  
week) it took me 30 minutes to build the tests and about the same to  
run the tests. I then ran it at least once a day (after I'd added/ 
changed things) to make sure I hadn't broken something.
>
>> Another problem here is that people may have different ideas on  
>> what  "Beta" means and I haven't seen it defined in terms of  
>> RunRev. One  company I worked for defined it as meaning "Feature  
>> Complete, No  Known Crashing Bugs".
>
> That's the ideal, but I've known no company that ships every Beta  
> in a state that meets that definition.
>

Well, I've beta tested Photoshop and I AFAIK there were no known  
crashing bugs and AFAIR it was feature complete.
> I've participated in tests for Adobe, Microsoft, Apple, Oracle, and  
> others who have shipped Betas while still developing new features.
>

"Feature Complete" was just the way that company did it, I've also  
seen that beta versions that still being developed. I was trying to  
find out what "beta" meant in the wonderful world of RunRev.

>
> That you're able to run stress tests on all features and identify  
> 100% of requirements to complete satisfaction before your first  
> Beta is quite an accomplishment,

Could you tell me where I said that? I run Stress Tests while  
developing my software as for requirements and when Beta testing is  
performed is up to the company I am working for in this case. In the  
past I have worked where this was the case though.

However stress testing is not an accomplishment at all it's really  
easy, that's why I really can't see why you are going on about it, I  
do it without even thinking about it! In fact I was absolutely  
gobsmacked that you were surprised that this was my way of working.

> and we look forward to URLs of the products you ship so we can all  
> learn how to improve the quality of our own work.

Now you're just being silly!  How will looking at the products I have  
worked on help you learn how to improve the quality of your own work?  
Besides which unless you just happen to have £50,000+ worth of video  
equipment lying around it wouldn't do you much good!

And Apart from all that - I've told you how to do it already!

Take Care and All the Best
Dave