A better way to gauge software release readiness

Online surveys are, in theory, a great way to gauge software release readiness. One would think it would be easy to send a survey link to all of one’s beta customers, asking them to rate the overall stability and individual new features in a release.

Unfortunately, it’s not so easy.

Not because online surveys are hard to create or expensive. Sites like Surveymonkey make it easy to create online surveys for a very reasonable fee.

The problem is getting users to actually fill out the surveys.

My own experience is that only about five percent of beta users fill out online surveys.

Why does this matter? Practically speaking, statistical significance kicks in around 27 responses. (That’s the rule of thumb taught in market research classes. Feel free to dig into the math if you want.) Dividing 27 by 5 percent means you need 540 beta users if you’re going to get a reasonable amount of certainty around release readiness.

Getting a few hundred beta users is not easy, given the fact that beta timeframes are often crunched down to the bare minimum time, caught between engineering’s inevitable release slips and sale’s understandable desire to start selling the new product as soon as possible. (I’m not complaining about engineering here — software development is a hard activity to do, and even harder one to forecast.)

Even if you have 500 or more beta users, it make take a couple of weeks to get to 27 responses. 15 might come the first week, 10 the next, and so on.

This slow accumulation of responses makes life hard for a software product manager. Every day that goes by is lost revenue, but you don’t want to pull the trigger and go GA without having met your release criteria.

How does a product manager address this?

The easiest way is to improve survey response rates. If you can get response rates to 10%, you only 270 beta users to get 27 responses. If you can get response rates to 30%, you only need 90 beta users.

One way to do this is with a contest — for instance, raffle the hot gadget of the day (iPod, iPhone), or give away a free license of your product.

Another, cheaper way is to use "inline surveys" that appear  right on your company’s home page or blog or some other highly-visited web page.

If beta users see a blog entry with a short, five question survey, they’re fairly likely to complete that survey because they know it won’t take up much of their day. If they see a Surveymonkey link, they have no idea how many questions are involved, and they bail.

This is why I’m excited about a new inline survey from a company called Vizu. They let you embed a polling widget into a web page with a bit of JavaScript. Their widget is technically the same as other kinds of web widgets that people put on their blogs and Facebook pages, and fairly easy to setup. Pricing starts at free.

[Disclosure: I'm friends with Dan Beltramo, Vizu's founder.]

Digg-style product feature voting

One of the toughest problems every software product manager faces is how to prioritize all the feature requests submitted by customers, salespeople, executives, and other stakeholders.

It’s a hard one to address. A software product can easily have hundreds, if not thousands, of feature requests. Companies often use bug tracking software, or support incident software, to track feature requests. Unfortunately, these don’t provide any way to "connect" requests. That is, to state that request A is the same as request B, and start tracking which requests are the most popular.

There are requirements management applications from FeaturePlan and Accept that let you connect feature requests. However, a product manager (PM) still has to manually make these connections.

A dirty secret in the product management world is that product managers (PM’s) rarely, if ever, comb through these feature request repositories when formulating a product roadmap. It simply takes too much time and there are too many other demands on one’s time.

The upshot is that when a new release comes out, it’s very easy for people to second-guess the product manager. Sales will ask, "why is feature X not in the release?" A big customer will ask, "why is my requested feature not implemented?"  Executives will ask, "is the PM really on top of customer needs?"

These kinds of questions, especially from executives, can hurt a product manager at review time. This is unfortunate, since PM’s typically do everything they can to provide a fair product roadmap.

This is why I’m very excited to learn about a new software application that lets stakeholders vote on product features. This is similar to how Digg lets users vote on stories, and how Dell Ideastorm lets users vote on feature ideas.

The application is called Pligg. It’s available for free download, and is hosted by several companies.

If usage of Pligg takes off among software companies, it could help our industry build much better products — and enable product managers to do much better jobs in less time.

I haven’t played with it yet, but am looking forward to doing so.

Have you used Pligg? Do you know of any other similar applications? Please let me know via the comments.

Can Twitter streamline Agile Development?

I’ve been thinking about if and how Twitter can be used by businesses to streamline their operations. Specifically, for software businesses, which is where my experience lies.

It seems to me that it can. Here’s one way:

Product development teams can use Twitter as an an "accelerator for Agile Development". Let me explain. One best practice of Agile Development is to a have a daily meeting, called a scrum. The meeting typically lasts ten or 15 minutes. It’s very informal. Every member of the development team briefly explains what they’re working on that day, and any blockers they face.

For instance, a developer might be waiting for product management to clarify a requirement. Or a tester might be waiting for a developer to check in a software component.

It sounds simple, but daily scrums are very effective in making problems known to the entire team, and consequently, getting problems fixed faster. With scrums, a blocking problem is unknown for only 24 hours max.

Now, what if a product team used Twitter to maintain a "continuous scrum status stream" — or scrum stream for short? Update your Twitter status once a day to declare what you’re working on. Update your Twitter status whenever a blocking issue comes up.

You would still need a daily scrum meeting, of course, to give everyone a live forum for raising issues. To be clear, I’m proposing Twitter to be used as a way to complement, not replace, existing Agile practices.

I think there would be at least two benefits to this use of Twitter.

Product team members would know about blocking issues even quicker. Problem resolution time would drop.

And, scrum meetings would be shorter, since they’d focus on exceptions, not day-to-day reporting of status.

In terms of security, Twitter already lets you restrict you can see your updates. So, if an engineering adopted Twitter to provide scrum streams, they would not have to worry about everyone on the web seeing those messages.

What do you think? Are there other ways you think Twitter can be used to make businesses work better?