Open Letter to Rev: Quality Is Job #1

Bill Marriott wjm at wjm.org
Fri Oct 20 17:46:42 EDT 2006


This post is much more constructive than your last one; it adds to the 
discussion and I appreciate it.

>> - The installer introduced in 2.7 is horrible. It installs desktop 
>> shortcuts that point nowhere. The file associations are busted.
>
> The team knows this (because someone posted it to Bugzilla) and is working 
> on it.

It's good that you admit to one problem, creating a broken desktop shortcut. 
Now, my question is: How could a problem like this have "slipped through the 
cracks?" It's only the number one thing someone does after downloading and 
running the Rev installer. Now, if you say a problem like this is invisible 
to the Rev team (it's existed since 2.7.1) then it begs the question of what 
kind of regression process is used prior to release. Perhaps you could allow 
that other glaring problems might escape notice as well?

> I have experienced none of the problems you report with instability, so 
> there is apparently an issue with something specific you are doing in your 
> stacks that the rest of us don't see. Again, a bug report that could 
> reproduce the problem would be welcome. Crashes are fixed first, with top 
> priority.

Or it could be with the fact that I have a dual processor. Or something else 
entirely. As I'm sure you know, random crashes are perhaps the most 
difficult problems to reproduce and pin down. No, I don't have a test case 
to trigger one and I haven't noticed a pattern. If I do, I will submit a 
report. My main issue is with the overall robustness of the product. I can 
work with confidence in 2.6.1, provided I save periodically. In 2.7.x, I 
don't have that confidence.

>> Half the time, quitting the standalone leaves the process running in Task 
>> Manager -- when the exact same stack under 2.6.1 exits cleanly.
>
> This, I believe, is a scripting issue on your part. Executables will not 
> quit if a running process has not completed; this is documented behavior. 
> I have never seen Revolution leave itself running in any of the stacks I'm 
> writing under Windows, so I have to assume that you are leaving something 
> open.

I do know that one has to unload speech, close pending messages, resolve 
Internet library requests, and so on. I don't generally use shell commands. 
I can say that a stack that always opens and quits cleanly with 2.6.1 can be 
opened in 2.7.4, built as a standalone, and run -- and will often NOT quit 
cleanly. Now, this is not 100% proof that the problem is not with my code. 
But, it does beg the question of what has changed to expose a problem.

Furthermore if it were just this one stack, then I'd again look to my own 
code for the cause. However, it happens in the IDE. Such as when I'm just 
launching it to write a "quickie" handler or test something out from the 
discussion list, or make minor edits to a stack. And it's not just leaving 
the process running... it's the whole universe of bad behavior that includes 
suddenly disappearing, quitting with a dialog, quitting with Windows Error 
Reporting, etc.

I do realize that it's possible for poor scripting to cause Rev to continue 
to appear in the task manager. However, it should not be allowed that a 
scripting error should cause a crash. And the IDE certainly shouldn't 
disappear during simple editing operations.

>> And Bill V's recent post is not the first report of standalones being 
>> built incorrectly or bugs returning from the dead.
> Perhaps, but I am more inclined to suspect that the wrong standalone 
> engine crept into his Runtime folder somehow. No one else has mentioned 
> any problems with it.

How exactly does the "wrong standalone engine" creep into the RunTime folder 
if the installer is working as it should? No I haven't personally 
experienced this -- I haven't invested enough time in 2.7.x -- but I'm not 
surprised, either. I would characterize the problems I have with 2.7.x as 
"blocking" issues. They prevent me from getting work done. Basically the 
routine is this:

- Rev announces a new version
- Filled with hope, I download and install it
- I groan as the installer fails its job miserably, and manually fix the 
shortcuts and file associations
- I launch Rev and check out the script editor and go get coffee as the 
system laboriously inserts a new blank line into my on mouseup handler (ok, 
that's an exaggeration, but it's damned slow, not even up to "notepad" 
standards).
- I try out a few sample scripts, such as trying a new feature or command
- If the IDE hasn't crashed, I try converting one of my working 2.6.1 
projects
- If things don't immediately go haywire I try to do some serious work

And inevitably, as I'm "getting into" the serious work part, things become 
less and less stable. Which leads me to the final steps:

- Giving up on the "latest and greatest"
- Saving any work in "legacy" format
- Uninstalling the new release
- Manually cleaning up the mess left behind
- Going back to Rev 2.6.1, where I don't experience these troubles

Now, what you're saying is that I should put several more hours of work into 
researching what specifically causes these problems so that I can file bug 
reports that may never get attention, for the honor of improving a product 
that I'm not going to purchase until it gets much, much more stable? What 
other industry works that way?

Have you guys forgotten who pays the bills? Ultimately, I have a program 
that works (2.6.1) and I'll keep using that until I see a release that earns 
my update fee.

>> When the Windows Error Reporting dialog comes up I click "Send Report" 
>> every time... do you guys even receive/read those reports? Because I 
>> haven't seen any decrease in their frequency from one release to the 
>> next.
>
> Runtime doesn't get these reports. The only way to be sure that the team 
> sees your report is to enter it into Bugzilla. That is the one and only 
> venue that assures the problem will be noticed.

Just an example of the Rev team limiting their awareness ("we don't read the 
listserv") at the expense of product quality. If Rev doesn't use WER that is 
disappointing. I can't think of a good reason why they wouldn't. As my 
previous post explained it is a FREE service that can dramatically speed 
identifying problems that cause crashes. The first link has information 
about the benefits of WER and how to register with Microsoft for the 
program. I hope Rev will take advantage of this tool so they can release 
software that is more stable.

> If you hate Bugzilla and can't stand to take the time to enter a report, 
> then write one up in your own words and send it to [support] along with 
> any stacks that reproduce the issue, and I will enter it into Bugzilla for 
> you. You will not get progress reports if you do that though; Bugzilla 
> update notices are sent only to the original reporter.

I don't hate BugZilla. I've used it before and entered bugs before. I've 
also searched BugZilla when I've encountered problems. It's not the 
prettiest thing in the world, but I know how to use it. The issues are 1) 
reproducability 2) confidence it will be addressed/fixed and 3) time. You 
make it sound as if I am somehow a lesser human being because I don't take a 
weekend or two to track down every problem I encounter with Rev's latest 
trial software. Sorry, but I don't think it's my job. I'm pretty busy as it 
is with work, friends and family. At the end point of the process above I 
already feel like I've wasted hours and fallen behind on whatever project I 
was trying to accomplish with Rev. You are perhaps compensated for the time 
you spend working for Rev; I am not.

> There are lots of people working with lots of copies of Rev and they don't 
> have these problems either. The only logical conclusion is that your 
> stacks do something that causes these issues [...]

No, that's not the only logical conclusion. One possibility is that existing 
users have already modified their behavior to avoid problems. Many on the 
list have boasted of using alternate IDEs, for example. Another is "learned 
helplessness" a well-known phenomenon where people realize there is nothing 
they can do about a situation, so they give up trying. A third possibility 
is that they care even less than I do about it -- or are a lot busier -- and 
just chuck it and move on. You only know what people report. Until my post 
you did not know I had a problem. If another user had written a similar post 
and you answered, "they don't have these problems" you would be incorrect. 
Only a fraction of those who use revolution keep up on the use-rev list. And 
not everyone writes support with their troubles.

I'm certainly not the only person who has stuck with 2.6.1. I wonder why 
they aren't updating?

As I tried to explain before, it's not consistent with human psychology to 
put more energy into something that seems futile.

> I had a similar experience about a year ago. I found the debugger to be 
> entirely unusable. It simply would not work under any circumstances, and 
> the internet library also failed. I was at a loss, and could not identify 
> the problem, but no one else had any trouble with it.
>
> Months later, I finally was able to pin down the fact that the debugger 
> didn't work if the stack file path had a comma in it. This turned out to 
> be the exact same problem that affected the internet library. I wrote to 
> Dave Cragg and he had a fix for me within a single day. I reported the 
> same thing to the team and the next release of Revolution had a fix in it.
>
> The reason I'm telling you this story is because I want you to understand 
> that I was as frustrated and unhappy as you are -- but I was aware enought 
> to know I was the only one apparently affected. The debugger was 
> completely, irretrievably broken for me, and yet no one else complained. 
> The issue was easy to fix once I identified the cause, but completely 
> unreproducible by anyone else.

This is a good "success story" and I am glad that you mention it. I will 
point out that it WAS in fact a problem with Rev and not a problem with your 
coding, that required you to put significant energy into testing, and 
resulted in a straightforward, if not easy, fix. It was probably NOT the 
case that "no one else" had a problem with it. Someone may well have had the 
problem and pulled their hair our trying to solve it, gave up, or found some 
other workaround. Maybe it cost Rev a sale or two. Who knows? Studies 
constently show that 90% of people never complain to a vendor/manufacturer 
about a problem... but 80% of them will remember their bad experience and 
tell others about it.

How many hours total of your involvement did it take from start-to-end 
because of this bug in Rev?

>> [issues with filing bug reports]
> Yes, that's true. And if *you* can't provide a reproducible stack -- and 
> you are the one having the trouble -- then how can the team do it? 
> Obviously it worked for all their tests prior to release. What are they 
> supposed to do to find the bug that you can't provide an example of?

I'm simply saying the caliber of the problems in Rev 2.7.x -- most glaring 
of which is the installer/uninstaller that doesn't work -- should be caught 
before end users see them. That they are not seen is indicative of a deep 
disfunction in Rev's testing/release process and should be addressed before 
end users will take it seriously.

>> Besides, I don't even "own" this version.
> That doesn't matter. You can submit to Bugzilla even if you are just a 
> trial user.

Sorry, my trial is expired now and I'm back to 2.6.1. Maybe I'll look into 
it again with 2.7.5.

I've got a suggestion, though I doubt Rev will take me up on it:

Put a bounty on bugs. Offer people a 1-month extension to their update pack 
if they find a confirmed level 3 bug. A 3-month extension if they find a 
confirmed crasher. If they're a trial user, give them a license for the 
version that fixes their bug. That will encourage users to submit thorough 
bug reports and for Rev to release cleaner code. 






More information about the use-livecode mailing list