[revServer] process timeout issue

[Kevin Miller]

On 04/08/2010 02:40, "Jeff Massung" <massung at gmail.com> wrote:

> I've tried to hold back from chiming in on yet another "passionate"
> discussion having to do with Rev. Most everyone here is great, and obviously
> there are those here intent on trying to diagnose and fix the problem.
> Bravo.

Hi Jeff,

Thanks for this helpful post. I'm going to take a few minutes out to respond
because I think its important.

> Yet, I feel the need to say that I had *extremely* similar issues to Jerry
> for the few months I used On-Rev.

I'm certainly sorry to hear that and we will continue to work with you to
try to identify whatever the cause is if that is what you want.

Not to take away from your experience or say that we won't continue to look
at ways of addressing it, I think its important to throw out a few data
points so readers get a more balanced picture of the situation.

There are many, many reasons why you might be experiencing problems with a
hosting company. Those problems range from possible issues with a technology
(e.g. in this instance revServer), to routing problems with our or your ISP,
to other users on the server having runaway processes that need to be better
capped, and so on. Then there is the coding that you or any customer on a
service like ours does and the possibility of there being some bug in that
which causes an issue. Like any complex computing problem with millions of
variables, the list of possible causes is virtually endless. Because we do
not know what Jerry's problem is, we cannot logically state that your
problem is the same problem. All that we can state is that we have a small
handful of customers, you and he included, who for whatever reason, have had
a performance problem. All our customers are important and these issues need
to be looked at. But at the same time, I doubt there is a hosting company in
existence that doesn't have a percentage of customers who experience a
problem at some time.

As CEO its my job to know whether or not we have an issue that affects a
small number of customers of if it affects lots of you. And I do have that
data at my fingertips. In our last customer survey, on-Rev had just over a
90% customer recommendation rate. That's pretty outstanding in this
industry. Even more so considering we are behind where we intended to be in
rolling out the finished revServer technology.

It is my belief that there are several good reasons for that outstanding
statistic. We have thousands of sites that are working perfectly. Absolutely
spot on, outstanding performance. Why do the perform so well? Because unlike
other shared hosting providers, we keep the overall number of users and
utilization per server extremely low. Our utilization is less than 20% of
another mainstream hosting provider which has often come in for positive
commendation on this list. Pretty outstanding value really. Then we have our
customer service, which is working extremely well. We're a small company yet
we have managed 24/7 cover with virtually no hitches and we almost never
have a queue or a long lag time even for non-urgent queries.

> During this time, Heather and Co. were very responsive and - I think -
> understanding of the problem(s) I was facing, but in the end weren't able
> to accommodate me in the way I was hoping they would. And that's 100% okay;
> no company can meet every customers' needs.

That's certainly fair enough. We tried, it didn't work out, it may not be
optimal but it can happen.

> It's really great that Rev users love their tool of choice. It's even better
> that they want to make it the best it can be. Sometimes I wonder if the Rev
> team realizes just how good of a marketing resource they have in their
> community. I rarely witness it being coddled, nurtured, loved, and even
> taken advantage of (in a good way). If I had such a zealous group behind one
> of my products, I'd be here every day trying to grow it.

I think its fair to say that we do a better job in direct interaction with
customers using our support at runrev.com address than on the mailing lists.
I've done a lot of reflecting on the business in the light of recent events
and I agree that this deserves to be addressed. We most certainly do
appreciate our many loyal customers - we are under absolutely no illusion
about that. They are truly a massive, primary motivation for all of us here
at RunRev. I would not be here writing this if that wasn't the case.

> But, in the end, we're customers. We're not paid advocates - at least I'm
> not. If a product I'm paying for isn't working for me, I move on. And - from
> what I've read and discussions had - that very much seems to be what Jerry
> did with On-Rev. We (and our business ventures) don't have infinite amounts
> of time and money in the bank to wait for a product to mature meet our
> needs, no matter how badly we want it to. And, likewise, unless the Rev team
> is aware that there is a serious risk of losing it's [very loyal] customers,
> there's no incentive for them to do better.

I appreciate that Jerry wanted to move on, given the type of business he was
entering into I think there might be business considerations driving at
that. That hunch is supported by his somewhat dismissive response to my post
that centered around our *technical* support. For the record, this is the
technical test we asked Jerry for when we started to work with him to try to
address the issues:

About 6 weeks ago, Kevin Miller wrote:
> One way to find out if it is something the script is doing is to measure the
> execution time of the whole irev script (i.e. use the long seconds at start,
> and append to a log file at the end). If, after observing a latency issue,
> there are 'spikes' in this measurement it means that the culprit is a
> side-effect of the script, rather than the startup time of the script.

Its not a hard test to run, and he didn't run it, so logically we can
conclude that Jerry had decided to move on for some other (perfectly valid)
reason. But just to be clear about how seriously *we* took this issue: we
also did some of our own tests to see how we performed relative to PHP.

FYI I've also posted the test we ran when investigating this issue in a
separate email.

So we took the issue seriously and were quite willing to work on it. We
haven't reached a conclusion and we didn't get any further information so we
weren't able to take it further. There may or may not be a different,
specific issue in this instance. If there is I'm sure we'll find it during
the normal course of improving revServer as it gets closer to release. But
the balance of probability is that there isn't some far reaching, sweeping
issue.

> From the outside, the Rev team feels like the exact opposite. I see
> nearly-zero focus on anything. On-Rev is > 1 years old now and the IDE is
> still something I wouldn't have release to any customer - even as an early
> alpha. There's no way they use that tool in-house, because any programmer
> worth their salt would have screamed out loud and started making it better
> within 48 hours of being forced to use it. And, to distract from that
> product line, there's also RevMobile, RevServer, RevWeb, RevDesktop, and
> whatever the next pre-alpha $400 product is in the pipe.

Well, its certainly true that we've had a few months that outwardly might
have borne more of a resemblance to wondering in the wilderness than is
healthy. We had a very public problem with revMobile and the knock on effect
has clearly been significant. I've already posted about that in our blog and
in emails to individual customers. I don't usually disclose much information
about our internals for competitive reasons (we're a privately held company
and its generally prudent to take advantage of that). But I think I'll
venture a little more perspective in this instance in the hopes that it is
useful.

The bad news internally was that we ended up having to lose just over 1/3 of
the company as a result of the revMobile debacle. That had a number of
predictable knock on effects. We had to move the remaining team members
around. We had to alter our development plan. Some of the work we had done
was no longer relevant. Some of it had to be done differently to take proper
advantage of the new opportunities that presented themselves.

I'm sure that this is not the sort of news that anyone in the community
wants to hear. But I'm sharing that news so that the community can
understand that an event like that naturally placed an unwelcome delay into
the middle of projects that we were working on. But equally important I
think its important for everyone to understand that this is the extent of
the issue, and for me to put it in perspective, and to explain that a lot of
good things are coming out of this too.

There is nothing like a good crisis to focus the mind, to make you truly
step back and look at your business and what you're doing well and what
you're not doing well. And in fact I've had more than one crisis, indeed
several this year, both personal and professional (its been a year from hell
almost like no other I've experienced!). But you can either buckle under
that or you can find the positive. And the good news is that soul searching
phase is done. We've done a root and branch analysis of everything from top
to bottom and reached lots of positive, exciting conclusions. I'm not ready
to share all the results today but I can give you a few examples. First case
in point: its clear we need to aim to do less and do it better. We need to
focus on the core things that we only we can do and work to better harness
the wonderful support in this community in other areas. We should be out
there, leading and focusing the energy of this community more regularly and
more consistently. One practical example of how we're going to do that is
that shortly after 4.5 we'll be making the externals (revVideoGrabber,
revBrowser, etc) an open source project which we will manage. That's a win
for everyone: we can continue to fix bugs and add features, while anyone in
the community can do the same. We will also have more time to work on the
core engine. There are lots of other examples in the works. We need to get
even closer to our customers (I'm here at 11PM doing that). We can see ways
to provide better support and make faster progress - through a combination
of factors including better focus, new processes; all in all we've figured
out a great plan to do that on many levels. And another example: our bug
management process isn't working well for our customers, its going to go.

So yes, recent events hit us hard and that's had an impact on the community
which we wouldn't have wanted. Its been tough and it's certainly not an
experience I care to repeat. We all wish for more good than bad in life, but
you have to take what comes and deal with both. I know I've done some things
well and that there other things I still need to improve at. But we have
taken the time we need to regroup, rethink, to do our homework, to rip up
the sacred cows and figure out some new ways of being. We have the talent
and capability to pull it off and reinvent ourselves. And so there is a new
sense of progress and dynamism here in Edinburgh as we put it all into
practice. Code is getting churned out at a good pace again upstairs. And on
my floor we're busy working on the new web presence that will support much
of this. Its not had a chance to ripple out yet into the community yet, its
going to take us until we launch 4.5 and even a little beyond to get all
this done. I'm not going to fall into the trap of pre-announcing all the
details instead of taking the time out to prepare and polish them properly.
I hope you can bear with us a little longer and be around to see the
results.

Before I finish on this subject, I think a little historical context is
useful. I've been at the helm of this company for over a decade. Its been a
decade with lots of learning. I'm 31 now, I have a very different view of
the world than I had when I started. For all the ups and downs, I've
presided over the growth of our customer base by several orders of
magnitude. I've faced good times and bad times and stuck through them all.
We have taken a hit and you've all witnessed a slow down that no-one wanted.
We may just have lost a chunk company we didn't want to lose. But in
context, we still have a bigger company than we had a couple of years ago.
And we're more experienced, have more perspective and are more focused than
we've ever been before. There are some great opportunities ahead. Here we
are and we have many, many wonderful customers doing awesome things with our
technology every day. We will look back in two years time and see this for
what it has been: a blip that ultimately made us stronger.

> Damn, that was a long-winded way of saying, please cut Jerry some slack.
> He's just another paying customer, and if the service provided doesn't live
> up to his expectations or needs, he moves on. No one should be upset at
> Jerry for any reason at all. If there's anything to be upset about, it would
> be asking one's self, "why on Earth wouldn't Rev do *everything* in their
> power to make such a long-standing customer incredibly happy so he could
> keep advocating Rev and even be able to claim that Rodeo runs on Rev
> servers?"

I like Jerry, I like Rodeo and I'll continue to point people to his product
if that seems to be a suitable fit for projects that need to deliver in that
specific way to that specific domain. And its true that perhaps the recent
business issues on our end have had some sort of impact. But I do need to
set the record straight here: we were more than willing to work hard to make
it work technically for Jerry. And beyond our professional interest in him
as a customer, it was partly my strong relationship with him and thus
genuine personal interest in seeing him succeed that persuaded me to allow
him to use this list as free marketing for his project. I can remember what
it was like starting out and I can see his obvious enthusiasm as he gets
going. I've enjoyed watching that and its been one (of several recent)
inspirations as I look forward to our own refresh that we're cooking up with
4.5.

But at the same time I have a business to run, a responsibility to all of
you and I cannot allow anyone to take advantage of my personal goodwill at
the expense of the Rev platform. I take exception to the fact that rather
than focusing on his own numerous strengths or citing business reasons, he
decided to take a shot at us, and did that based on a technical, or
technical support issue. Given the nature of his product Jerry may well have
had legitimate business reasons for moving to PHP. And there may be some
specific issue in revServer that need work. But his remarks have left our
community scratching their heads to try to identify whether or not there is
some fundamental issue with the technology or with on-Rev. I have cited data
and experience to explain why I don't believe there is. And I do believe
that any specific problems that occur in limited circumstances will be
addressed by our outstanding engineering team who are very much back in the
game.

Kind regards,

Kevin

Kevin Miller ~ kevin at runrev.com ~ http://www.runrev.com/
RunRev - Software construction for everyone