[continued from Beta move on]
Standard treatment
In most cases, companies use closed beta testing, limiting and controlling the distribution of beta versions of the software. Finding and managing beta testers becomes an issue, and finding good testers is a difficult challenge, so we need to discuss the closed beta process in more detail.
The unfortunate fact is that few users know how to properly test software, so if you are lucky enough to find a good tester, make certain that you keep that person happy. Useful feedback should be rewarded with a free copy of the program, at a minimum, and the tester should always be invited to participate in future beta tests. A good tester will outperform a dozen mediocre testers and, therefore, is very valuable.
A related problem is that many prospective testers will not provide any feedback at all, so it is necessary to invite more beta testers than you expect to need. You can anticipate that roughly half of the beta testers in a closed beta will not report anything at all, and some of the others will not be useful. In most cases, it is difficult to find enough beta testers, so it is unlikely that a product will get too many volunteers.
When looking for beta testers, cast a wide net. It is important to have as large a range of experience levels, methods of use, and system configurations as possible. It is a good idea to ask potential beta testers not only for contact information, but also about system configurations and software experience.
Remember, some of your potential customers are likely to be struggling with computer illiteracy, so it makes sense to have some less experienced testers as well. Knowledgeable users will often figure out how to do something, or find a workaround, on their own without indicating that there may be a problem. Neophytes, on the other hand, will ask questions that customers would ask. Do not rely solely on other developers for testing unless your product can only be used by programmers.
The best means of communication for a closed beta process is beta forum of some kind, in which beta testers can interact with each other. This helps establish a sense of community that works to support tester involvement and breeds loyalty to the product. From a practical standpoint, this also allows problems to be independently verified by other testers, and they will often work together to help you replicate a bug. There should also be an email address for bug reports, but forum participation should be encouraged.
It is important to remember that beta testing is not an adversarial process. Let me say that again. Beta testing is not an adversarial process. It can sometimes be very difficult to take criticism, but you must be certain not to get defensive. Always wear a (virtual) smile. Beta testers are there to help you, and it is far better to hear about problems now rather than after release.
All feedback is beneficial, so you should listen to everything that is reported. Try to respond to every report so that testers know you are listening and involved, which gives a psychological incentive to do a better job. Avoid being dismissive, as that discourages participation. Also, make it clear that you appreciate the reports, even the negative ones, since some testers are reluctant to report bugs or bad impressions if they feel that you will be insulted. Many reports are preceded by apologies.
One technique for keeping testers involved is to provide means of communication that does not necessarily involve bugs reports. Informal surveys about aspects of the program or system hardware questionnaires give testers a change to participate even if they cannot find any bugs (which is the goal, after all). In my last beta test, I decided to try a little contest. I found three unreported bugs in different areas of the game and challenged the testers to find them. The number of valid bug reports increased measurably.
[continued in Something different]