Open Letter to Rev: Quality Is Job #1
Bill Marriott
wjm at wjm.org
Fri Oct 20 17:46:42 EDT 2006
This post is much more constructive than your last one; it adds to the
discussion and I appreciate it.
>> - The installer introduced in 2.7 is horrible. It installs desktop
>> shortcuts that point nowhere. The file associations are busted.
>
> The team knows this (because someone posted it to Bugzilla) and is working
> on it.
It's good that you admit to one problem, creating a broken desktop shortcut.
Now, my question is: How could a problem like this have "slipped through the
cracks?" It's only the number one thing someone does after downloading and
running the Rev installer. Now, if you say a problem like this is invisible
to the Rev team (it's existed since 2.7.1) then it begs the question of what
kind of regression process is used prior to release. Perhaps you could allow
that other glaring problems might escape notice as well?
> I have experienced none of the problems you report with instability, so
> there is apparently an issue with something specific you are doing in your
> stacks that the rest of us don't see. Again, a bug report that could
> reproduce the problem would be welcome. Crashes are fixed first, with top
> priority.
Or it could be with the fact that I have a dual processor. Or something else
entirely. As I'm sure you know, random crashes are perhaps the most
difficult problems to reproduce and pin down. No, I don't have a test case
to trigger one and I haven't noticed a pattern. If I do, I will submit a
report. My main issue is with the overall robustness of the product. I can
work with confidence in 2.6.1, provided I save periodically. In 2.7.x, I
don't have that confidence.
>> Half the time, quitting the standalone leaves the process running in Task
>> Manager -- when the exact same stack under 2.6.1 exits cleanly.
>
> This, I believe, is a scripting issue on your part. Executables will not
> quit if a running process has not completed; this is documented behavior.
> I have never seen Revolution leave itself running in any of the stacks I'm
> writing under Windows, so I have to assume that you are leaving something
> open.
I do know that one has to unload speech, close pending messages, resolve
Internet library requests, and so on. I don't generally use shell commands.
I can say that a stack that always opens and quits cleanly with 2.6.1 can be
opened in 2.7.4, built as a standalone, and run -- and will often NOT quit
cleanly. Now, this is not 100% proof that the problem is not with my code.
But, it does beg the question of what has changed to expose a problem.
Furthermore if it were just this one stack, then I'd again look to my own
code for the cause. However, it happens in the IDE. Such as when I'm just
launching it to write a "quickie" handler or test something out from the
discussion list, or make minor edits to a stack. And it's not just leaving
the process running... it's the whole universe of bad behavior that includes
suddenly disappearing, quitting with a dialog, quitting with Windows Error
Reporting, etc.
I do realize that it's possible for poor scripting to cause Rev to continue
to appear in the task manager. However, it should not be allowed that a
scripting error should cause a crash. And the IDE certainly shouldn't
disappear during simple editing operations.
>> And Bill V's recent post is not the first report of standalones being
>> built incorrectly or bugs returning from the dead.
> Perhaps, but I am more inclined to suspect that the wrong standalone
> engine crept into his Runtime folder somehow. No one else has mentioned
> any problems with it.
How exactly does the "wrong standalone engine" creep into the RunTime folder
if the installer is working as it should? No I haven't personally
experienced this -- I haven't invested enough time in 2.7.x -- but I'm not
surprised, either. I would characterize the problems I have with 2.7.x as
"blocking" issues. They prevent me from getting work done. Basically the
routine is this:
- Rev announces a new version
- Filled with hope, I download and install it
- I groan as the installer fails its job miserably, and manually fix the
shortcuts and file associations
- I launch Rev and check out the script editor and go get coffee as the
system laboriously inserts a new blank line into my on mouseup handler (ok,
that's an exaggeration, but it's damned slow, not even up to "notepad"
standards).
- I try out a few sample scripts, such as trying a new feature or command
- If the IDE hasn't crashed, I try converting one of my working 2.6.1
projects
- If things don't immediately go haywire I try to do some serious work
And inevitably, as I'm "getting into" the serious work part, things become
less and less stable. Which leads me to the final steps:
- Giving up on the "latest and greatest"
- Saving any work in "legacy" format
- Uninstalling the new release
- Manually cleaning up the mess left behind
- Going back to Rev 2.6.1, where I don't experience these troubles
Now, what you're saying is that I should put several more hours of work into
researching what specifically causes these problems so that I can file bug
reports that may never get attention, for the honor of improving a product
that I'm not going to purchase until it gets much, much more stable? What
other industry works that way?
Have you guys forgotten who pays the bills? Ultimately, I have a program
that works (2.6.1) and I'll keep using that until I see a release that earns
my update fee.
>> When the Windows Error Reporting dialog comes up I click "Send Report"
>> every time... do you guys even receive/read those reports? Because I
>> haven't seen any decrease in their frequency from one release to the
>> next.
>
> Runtime doesn't get these reports. The only way to be sure that the team
> sees your report is to enter it into Bugzilla. That is the one and only
> venue that assures the problem will be noticed.
Just an example of the Rev team limiting their awareness ("we don't read the
listserv") at the expense of product quality. If Rev doesn't use WER that is
disappointing. I can't think of a good reason why they wouldn't. As my
previous post explained it is a FREE service that can dramatically speed
identifying problems that cause crashes. The first link has information
about the benefits of WER and how to register with Microsoft for the
program. I hope Rev will take advantage of this tool so they can release
software that is more stable.
> If you hate Bugzilla and can't stand to take the time to enter a report,
> then write one up in your own words and send it to [support] along with
> any stacks that reproduce the issue, and I will enter it into Bugzilla for
> you. You will not get progress reports if you do that though; Bugzilla
> update notices are sent only to the original reporter.
I don't hate BugZilla. I've used it before and entered bugs before. I've
also searched BugZilla when I've encountered problems. It's not the
prettiest thing in the world, but I know how to use it. The issues are 1)
reproducability 2) confidence it will be addressed/fixed and 3) time. You
make it sound as if I am somehow a lesser human being because I don't take a
weekend or two to track down every problem I encounter with Rev's latest
trial software. Sorry, but I don't think it's my job. I'm pretty busy as it
is with work, friends and family. At the end point of the process above I
already feel like I've wasted hours and fallen behind on whatever project I
was trying to accomplish with Rev. You are perhaps compensated for the time
you spend working for Rev; I am not.
> There are lots of people working with lots of copies of Rev and they don't
> have these problems either. The only logical conclusion is that your
> stacks do something that causes these issues [...]
No, that's not the only logical conclusion. One possibility is that existing
users have already modified their behavior to avoid problems. Many on the
list have boasted of using alternate IDEs, for example. Another is "learned
helplessness" a well-known phenomenon where people realize there is nothing
they can do about a situation, so they give up trying. A third possibility
is that they care even less than I do about it -- or are a lot busier -- and
just chuck it and move on. You only know what people report. Until my post
you did not know I had a problem. If another user had written a similar post
and you answered, "they don't have these problems" you would be incorrect.
Only a fraction of those who use revolution keep up on the use-rev list. And
not everyone writes support with their troubles.
I'm certainly not the only person who has stuck with 2.6.1. I wonder why
they aren't updating?
As I tried to explain before, it's not consistent with human psychology to
put more energy into something that seems futile.
> I had a similar experience about a year ago. I found the debugger to be
> entirely unusable. It simply would not work under any circumstances, and
> the internet library also failed. I was at a loss, and could not identify
> the problem, but no one else had any trouble with it.
>
> Months later, I finally was able to pin down the fact that the debugger
> didn't work if the stack file path had a comma in it. This turned out to
> be the exact same problem that affected the internet library. I wrote to
> Dave Cragg and he had a fix for me within a single day. I reported the
> same thing to the team and the next release of Revolution had a fix in it.
>
> The reason I'm telling you this story is because I want you to understand
> that I was as frustrated and unhappy as you are -- but I was aware enought
> to know I was the only one apparently affected. The debugger was
> completely, irretrievably broken for me, and yet no one else complained.
> The issue was easy to fix once I identified the cause, but completely
> unreproducible by anyone else.
This is a good "success story" and I am glad that you mention it. I will
point out that it WAS in fact a problem with Rev and not a problem with your
coding, that required you to put significant energy into testing, and
resulted in a straightforward, if not easy, fix. It was probably NOT the
case that "no one else" had a problem with it. Someone may well have had the
problem and pulled their hair our trying to solve it, gave up, or found some
other workaround. Maybe it cost Rev a sale or two. Who knows? Studies
constently show that 90% of people never complain to a vendor/manufacturer
about a problem... but 80% of them will remember their bad experience and
tell others about it.
How many hours total of your involvement did it take from start-to-end
because of this bug in Rev?
>> [issues with filing bug reports]
> Yes, that's true. And if *you* can't provide a reproducible stack -- and
> you are the one having the trouble -- then how can the team do it?
> Obviously it worked for all their tests prior to release. What are they
> supposed to do to find the bug that you can't provide an example of?
I'm simply saying the caliber of the problems in Rev 2.7.x -- most glaring
of which is the installer/uninstaller that doesn't work -- should be caught
before end users see them. That they are not seen is indicative of a deep
disfunction in Rev's testing/release process and should be addressed before
end users will take it seriously.
>> Besides, I don't even "own" this version.
> That doesn't matter. You can submit to Bugzilla even if you are just a
> trial user.
Sorry, my trial is expired now and I'm back to 2.6.1. Maybe I'll look into
it again with 2.7.5.
I've got a suggestion, though I doubt Rev will take me up on it:
Put a bounty on bugs. Offer people a 1-month extension to their update pack
if they find a confirmed level 3 bug. A 3-month extension if they find a
confirmed crasher. If they're a trial user, give them a license for the
version that fixes their bug. That will encourage users to submit thorough
bug reports and for Rev to release cleaner code.
More information about the use-livecode
mailing list