Online surveys are, in theory, a great way to gauge software release readiness. One would think it would be easy to send a survey link to all of one’s beta customers, asking them to rate the overall stability and individual new features in a release.
Unfortunately, it’s not so easy.
Not because online surveys are hard to create or expensive. Sites like Surveymonkey make it easy to create online surveys for a very reasonable fee.
The problem is getting users to actually fill out the surveys.
My own experience is that only about five percent of beta users fill out online surveys.
Why does this matter? Practically speaking, statistical significance kicks in around 27 responses. (That’s the rule of thumb taught in market research classes. Feel free to dig into the math if you want.) Dividing 27 by 5 percent means you need 540 beta users if you’re going to get a reasonable amount of certainty around release readiness.
Getting a few hundred beta users is not easy, given the fact that beta timeframes are often crunched down to the bare minimum time, caught between engineering’s inevitable release slips and sale’s understandable desire to start selling the new product as soon as possible. (I’m not complaining about engineering here — software development is a hard activity to do, and even harder one to forecast.)
Even if you have 500 or more beta users, it make take a couple of weeks to get to 27 responses. 15 might come the first week, 10 the next, and so on.
This slow accumulation of responses makes life hard for a software product manager. Every day that goes by is lost revenue, but you don’t want to pull the trigger and go GA without having met your release criteria.
How does a product manager address this?
The easiest way is to improve survey response rates. If you can get response rates to 10%, you only 270 beta users to get 27 responses. If you can get response rates to 30%, you only need 90 beta users.
One way to do this is with a contest — for instance, raffle the hot gadget of the day (iPod, iPhone), or give away a free license of your product.
Another, cheaper way is to use "inline surveys" that appear right on your company’s home page or blog or some other highly-visited web page.
If beta users see a blog entry with a short, five question survey, they’re fairly likely to complete that survey because they know it won’t take up much of their day. If they see a Surveymonkey link, they have no idea how many questions are involved, and they bail.
[Disclosure: I'm friends with Dan Beltramo, Vizu's founder.]