'Community Beta' has lost its way

[Bill Marriott]

Bernard,

I'm glad you brought up the post I made last year, because it reminds me of 
how far we've come since then.

Not only did we implement a substantial update to BugZilla, making it far 
more accessible and effective, but the effort going into BugZilla entries is 
like night and day as well. It's not just me trying to put the best face on 
things, either. The surveys we have conducted prove it.

As a beta tester, you know we are conducting regular surveys to assess the 
progress we're making against the quality objective. Here are actual results 
from the survey question regarding bug reporting:

#14) Please indicate the degree to which you agree with the following 
statements about BugZilla/the new Revolution Quality Control Center:

-100 = Strongly Disagree
 0 = Neither Agree nor Disagree
+100 = Strongly Agree
(average ratings)

I am able to enter my reports accurately and completely
12-2006: +37
05-2007: +56

The reports I make are read/reviewed promptly
12-2006: +8
05-2007: +33

I receive adequate feedback on my reports
12-2006: +10
05-2007: +34

Runtime Developers take my reports seriously
12-2006: +14
05-2007: +40

The voting system is a good way to prioritize bugs
12-2006: -6
05-2007: +16

The bugs I care about are being, or have been addressed
12-2006: -12
05-2007: +14

(The second beta tester survey is not closed yet, but we've already reach 
80% of the response rate we're looking for, so the numbers are not going to 
change that much.)

The results above speak for themselves. They are evidence there has been 
real and dramatic progress made against the objective of making 
BugZilla/RQCC a valuable, credible resource for improving the quality of 
Revolution, and that RunRev developers have been more responsive to user 
needs. Additionally, we've added more than 100 users of the system since the 
"Open Beta" program launched, and they have been filing even more bugs as a 
result. We've had more users, more bugs filed, and yet better engagement 
from Runtime Revolution than ever before. No small accomplishment. That's 
not marketing spin, that's the truth. Is there room for improvement? Of 
course. But as more bugs are fixed, I expect these numbers to improve even 
more.

It is absolutely worthwhile now to use the Revolution Quality Control Center 
to report your issues. This is a decided change from when I made my October 
post.

> That's exactly how I feel.  Only the difference is that you now bear 
> responsibility with regard to this.  Bug 3196 went through the proper 
> bug-reporting procedure (and in fact also had duplicate postings).

Bernard, I am sorry that your particular issue wasn't addressed in the two 
weeks since you sent me your mail, or the few months since you first 
reported it. However, it is being studied, it's marked and logged as a "Top 
5" issue, and not being ignored. I do hope it will be addressed for 2.9.

> I cannot understand how this old, blocker-level bug is still in the 
> Release Candidate, nor can I understand how Rev's Quality Control  manager 
> does not visit  the Community Beta Forum.  Maybe that forum  would have 
> more activity if people actually saw that you visited it  regularly. 
> Might I suggest that you post an announcement there  saying that people 
> should email you directly instead of wasting their  time in that forum? 
> Mind you, I did email you directly twice, and  it made no difference to 
> this bug's status.

Like many people, I find the mailing lists and RQCC to be a much more 
efficient way to exchange information. The forum is simply not trafficked 
heavily in general, and I do not receive notifications reliably when the 
occassional post is made. Having said that, there is a decent amount of 
material there which help people understand the bug reporting process, gives 
testing tips, and helps people write better bugs. I try to stop in there 
periodically, but honestly it's not a top priority -- it's supposed to be a 
user-to-user forum. As for the email, as I've explained to you I requested 
people send me bugs. It's was not my intent to open an email exchange on 
each and every one of them. I requested people send me a bug; you sent me 
one; it was logged and recorded. I didn't even realize you expected a 
personal response, and if you had written me an direct question or request 
for information you would have gotten one.

> From what you are saying, I must be in a minority.  In one sense,  I'm 
> glad to hear that.  On several occasions I've suggested that  there be  a 
> list of the bugs that were fixed on each iteration of the  beta cycle. 
> That would have enabled people to see that even if bugs  that concerned 
> them were going unfixed, that other bugs were still  being fixed.

The list of bugs fixed is in the RQCC itself. The status messages generally 
are a good indicator what is being worked on. If it's "unconfirmed" then it 
hasn't received a lot of attention yet. If it's "new" them people know about 
it and it's being slated. You can easily create a saved search to indicate 
which bugs have changed recently, or to report on the latest comments by the 
engineering team. RunRev could and should make better use of RQCC as a tool 
for communicating status, but I think they're well on their way and steadily 
improving.

> Each time I downloaded the beta, uninstalled the old version,  installed 
> the new version, then I would run my tests to see if bug  3196 had been 
> fixed (even though it wasn't listed in any of the  accompanying "Change 
> Log.txt" files).  Each time I'd be  disappointed.  Eventually it dawned on 
> me that it wasn't going to get  fixed.  When I saw it was still there in 
> the Release Candidate, and  there was no response from anyone at 
> runrev.com about my warnings, I  realised something was seriously awry.

Unfortunately, we do have release candidates and "golden masters" with open 
bugs. That is a fact of life in the software world in general. Issues are 
prioritized by a number of factors, including the votes received, difficulty 
of researching/fixing them, severity of the bug, number of people it 
affects, business concerns, and so on. Report #3196 has 20 votes for it so 
far. That means it's up there on the list, but by no means in the top 50%.

I suggest you count the number of open bugs, then make a guess as to how 
many hours it will take to reproduce, research the code, write a fix, and 
test all of them thoroughly. Do the multiplication and you'll find there are 
more hours than left in the next year (or two). This is why I asked for the 
"Top 5" bugs from people, and why we have the voting system.

> So as far as I'm concerned the Community Beta has clearly lost its  way. 
> It promised to focus on removing serious bugs and introducing  the 
> long-awaited Linux version.  Instead serious, documented bugs  have been 
> allowed into the Release Candidate, the Linux version has  been pushed 
> out, and the emphasis seems to have turned to introducing  new features.

It may not be moving as fast as you'd like, but it is going in the right 
direction. There are a large number of open bugs in the system. Hundreds --  
literally hundreds -- are being fixed with every release. It's going to take 
a long time to reach perfection. I expect Revolution 2.9 will be extremely 
welcomed and largely satisfy the goals of a quality release. In the 
meantime, I don't think it's reasonable for Revolution to stand without a 
single release until your bug -- or anyone else's personal issue -- is 
fixed.

> You can harp on about the incorporation of the Altuit products, but  they 
> were not initially part of the Community Beta.  That's a red  herring.  Of 
> course those products have to be incorporated at some  point - they were 
> probably bought as a band-aid to give Rev added- value, whilst your 
> attempted consumer revolt over bugs was managed.   Since the Altuit 
> externals had obviously been working fine for  Altuit's customers, there 
> was no need that they be seamlessly  introduced in this Beta cycle.

I think the "new features" as such have been modest and absolutely 
necessary. Take the Altuit externals. Rev released those as a separtae 
download shortly after they were acquired. Nevertheless, the documentation 
was sparse and all over the place. The original sample stacks worked okay 
when they were distributed by Altuit but didn't make sense when it was 
coming from RunRev. The Font external was busted on Mac OS Intel. There was 
a senseless "activation" business where you had to put a serial number into 
your code. And having the separate download had a half-baked feel to the 
process overall. It was very difficult for a new user to understand how to 
make use of them. It's a good thing they are integrated, and have several 
bugs fixed in them now. As for the functionality they provide, I consider 
them pretty essential.

> Remember: Quality is Job #1

The unwavering focus since November has been on improving the quality of the 
product, in all the forms that quality takes. Sorry if we disappointed you 
regarding this particular issue, and I know your support of Revolution is 
appreciated. We can disagree about whether a typo in the docs is more or 
less important than a database plugin or is more or less important than the 
sockets issue you reported. But I am satisfied that quality is indeed job #1 
now at RunRev.

I'll close with some actual quotes from the latest survey:

"The pre-release program is getting better over time."
"Greatly Impressed with the progress made"
"It is very nice to see how fast the bugs I submitted are fixed."
"haven't felt nervous about using [the betas] for virtually all my 
development work. They all felt more stable than 2.7.x."
"I am super thrilled to have such a solid release coming to us. 2.8.1 feels 
to be the best version of Rev ever."
"Very stable. Light night and day"
"Progress is being made in the right direction."
"This is by far the best release of Rev yet. Lots of improvements that make 
my apps look/behave more professionally."
"A very good experience for me, if not for you, which has decided me to 
learn again to use Revolution."