Quality: The Process, Part III [Beta move on]

[continued from Greek to me]

Beta move on

When the program is feature complete, or approaching that stage, it is time to consider taking the next step. One step from alpha is beta, so we should now look at “beta testing”.

Beta testing is the most recognized form of black box testing, in which the software is submitted to users outside the company for additional testing and feedback. Generally, these testers are not professionals, but rather should represent a typical cross-section of potential customers and users.

Since beta testing is often the first external exposure of your product, it is important that the alpha testing and glass box techniques have produced a reasonably solid program. It may be a cliché, but there is not a second chance to make a first impression. When a tester’s first experience with a product is lousy, he or she will be less likely to get comfortable with it. If you know that there are lots of bugs, then your software is probably not ready for beta testing.

A practical reason for making sure the software already shows a standard of quality when beta testing begins is that obvious bugs will be reported multiple times, and less severe bugs will be overlooked. When a tester finds a number of problems, he or she may relax the reporting or assume that one bug is caused by another. Also, some bugs do cause a multiplicity of symptoms, and tracking becomes more convoluted.

There are two primary forms of beta testing, “open” and “closed”. In open beta testing, the developer announces the availability of a “public beta” version of the software, and any interested party can download and test the software. For closed beta testing, the developer provides a “private beta” version of the software to a limited number of known testers.

Companies may use either or both forms of beta testing. The main advantage of open beta testing is that the software can be tested by lots of people to cover a wide array of systems and uses, at the expense of control and a possible impact on the marketing plan. On the other hand, closed beta testing provides the developer with better control of the process, but the disadvantage is that it is hard to find testers.

Some companies use both forms of beta testing, starting with a closed beta and then expanding to an open beta program once the program is closer to release. Microsoft, for example, runs an extensive closed beta testing program for DirectX, including the SDK and the runtimes, which lasts for several months each version, but near the end of this process, the beta runtimes are made available for public download. [Note: Microsoft has since ceased proper testing of DirectX SDK releases and is now a counter example, not to be followed.]

For either form of beta testing, you should insert a “drop dead” date in the code, so the program will not run after a certain fixed date. This prevents the beta from entering general circulation and reduces testing of outdated versions. Note that this technique should never be used for release versions, so you must remember to remove it before the final version. You must also remember to update the date with each new testing version lest you have a valid beta timeout prematurely.

Just as a feature complete product signals the approaching end of the alpha testing phase, the impending completion of the beta testing phase is signaled by a “release candidate”. A release candidate is a version of the product that is potentially the release version of the software. At this point, testers should be instructed to report every bug they find, even if they have reported it previously, since all bugs should have been eliminated. If bugs are corrected, another release candidate should be created and tested.

For the first release of a product, the traditional beta version numbers start at 0.90 and approach 1.0, the release version. I know of one game product, on which I did not work, that had so many beta versions that the producer gave the team shirts that read “Version 0.99999999…” with the nines running all of the way down one of the sleeves.

[continued in Standard treatment]

Leave a Reply

Your email address will not be published.