[This article was originally published in the December 2002 issue of ASPects.]

“Come, give us a taste of your quality.”

When Shakespeare penned these words for Hamlet, he was referring to the calling of those to whom the comment was directed. In this context, quality is literally ones profession. As software developers who, for the most part, can either succeed or fail based on the quality of our work, this interpretation still holds true.

In the first part of this article, I discussed planning for quality software development, classification and tracking of bugs and unimplemented features, and some basic quality assurance concepts and terms. In particular, I explained black box and glass box testing, some methods for which we will explore in this part. As a simple reminder, black box testing is, essentially, user testing of the program itself, while glass box testing is developer testing of both source and object code.

You have been waiting long enough, so it is time to put these ideas into practice.

[continued in Development environment]

[continued from Bug and feature tracking]

More to come

To this point, we have chosen a project, determined what we want to accomplish with the end product, and proved in the design document that we can meet our goals with it. We have recommitted to developing a quality product and we have devised our own system for tracking the features and the bug reports that will inevitably be received, and some of us have even populated the software with features from the design docs.
In short, we have established a strong foundation for quality software development.

Now, all of the real programmers among us are jumping up and down yelling, “Let’s get to the coding already!” With this foundation, we are in a great position to get started with the actual programming. Unfortunately, the discussion of specific glass box and black box techniques, including beta testing methods, will have to wait for the next installment.

I have some coding to do.

Gregg Seelhoff is an independent game developer who serve[d] on the ASP Board of Directors. He can be reached at seelhoff@sophsoft.com.

[continued from Getting started]

Bug and feature tracking

Classification of discovered and reported bugs can be beneficial in determining priorities for development resources. High priority bugs need to be addressed as soon as reasonably possible, while suggestions should probably wait. In my experience, most companies, including Microsoft, use four or five classifications, similar to the following:

1. 1 – Severe error – This includes program crashes, errors which damage data or interfere with the operating system, or anything that prevents further testing. These types of bugs are known as “showstoppers” and, though hopefully infrequent, are urgent issues.
2. 2 – Functionality impaired – This includes any type of bug in which the program does not perform as expected and has a detrimental impact on the ability to use the program.
3. 3 – Minor issue – This includes bugs which are cosmetic, such as spelling errors, or those that do not significantly affect the ability to use the program efficiently.
4. 4 – Suggestion – This includes any suggestions for unplanned and non-essential features. These should generally be implemented only later in development, and changes to address these issues should be reflected in updated design documentation.
5. 5 – Postponed – This includes any suggestions or, in some cases, problematic bug fixes that are not likely to be in the current release, but that should nevertheless be retained for future consideration.

In parallel to the above classification of reported bugs, we also use a similar scale for prioritizing product features:

1. 1 – Essential – These are features without which the program cannot ship.
2. 2 – Important – These are features which should be in the program.
3. 3 – Desired – These are features that we want to be implemented, if possible.
4. 4 – Extra – These are features that would be nice to incorporate, if there is time.
5. 5 – Wish list – These are features that will probably have to wait for a later version.

Despite the fact that the two classification systems are similar, we generally prioritize bug fixes (levels 1-3) before implementation of new features. The only times in which a bug fix is postponed temporarily are when a feature implementation is incomplete and needs to be finished first, when an upcoming feature will obviate the bug, or when the bug is so minor that delaying the fix has no discernible adverse impact on the product.

As you begin development, it is important that you devise a method for tracking and managing both feature development and bug reports. Throughout the process, priorities invariably change and bugs will be uncovered through testing, and if you do development work for somebody else, new feature requests and change orders need to be handled. A good system will help keep the development in perspective and prevent issues from getting forgotten.

For tracking bugs and features, there are a number of software options, some of which are very expensive. Some ASP members produce reasonable software for tracking product features and bugs, so check the download site for some software designed for that purpose. I personally prefer a simple solution, so my feature and bug tracking is done with WordPad and one physical file folder. Another intriguing approach, which I learned from the member newsgroups, is to use a deck of index cards, one for each issue, and remove items physically when they have been resolved.

[continued in More to come]

[continued from Planning for Quality]

Getting started

As soon as program development commences, you can, and should, begin testing the software in various ways. Once you have adopted the attitude that producing a quality product is paramount, incorporating the testing process into development is the single biggest step that you can take to achieve the desired result.

At this point, it will be very useful to define some terms. The one known to most software users is “bug”, which refers to an unexpected problem with software (as opposed to a design decision or “feature”). The term comes from the early days of electronics, but discovery of the first computer bug is attributed to Grace Hopper, who, in 1945, extracted a moth from a relay that had apparently beaten it to death. She pasted it into a logbook, and now this original bug resides at the Smithsonian Institution, still attached to that same page.

In searching for bugs, there are, essentially, two different kinds of testing, known as “black box” and “glass box” (or “white box”) testing. Black box testing, as the name implies, is testing which treats the software as a black box, checking only the output based on the provided input. Glass box testing, on the other hand, is testing which makes use of special knowledge or access to the inner workings of the software.

Black box testing is the most common form of software testing, and is what most people mean when they use the word, “testing”. One simply runs the software to see if it does what it is supposed to do. If the program does something incorrect or looks wrong, then there is probably a bug, and with some luck, the developer will get a defect report and be able to fix the problem.

Glass box testing methods can significantly decrease the number of software defects and improve software quality. This is especially true in development environments in which glass box testing has not been used previously. There are a number of glass box techniques that can be utilized as development is underway to help identify and eliminate bugs before they reach the black box stage.

Another couple of terms that are commonly used with regard to testing are “internal” and “external”. Internal testing is any form of testing in which the process is conducted and controlled by the developer, without distribution of the product in any form. External testing is any form of testing in which the product is distributed, whether widely or in a limited way, for testing purposes.

[continued in Bug and feature tracking]

[continued from Quality: The Process, Part I]

Planning for Quality

The quality process begins with the initial product concept. From its inception, the development of a product must be planned. Basically, if you do not know where you are going, it is unlikely you will get there. If you do not know the route, you may get lost along the way and probably will not get to your destination efficiently. Neither situation is conducive to quality software.

The tools for helping software development maintain the correct direction throughout a project are design documents. Design documents should do exactly what they say: document the design of a program. The actual contents and approach vary greatly depending on the type of software and the author(s) of the document, but they should provide a written roadmap for the project.

One very important aspect of design documents, and a reason why even lone developers should use them, is that they provide the opportunity to discover flaws in the planning before time is wasted on unproductive activities or projects. When one is forced to document the design, one must think through all aspects of the project. If the project is not viable, for whatever reason, it is better to find out early, rather than too late.

From a quality perspective, the most important aspect of a design document is what is known as the technical specification, which describes in detail the technical aspects of the project, including the program interface and any complex procedures. During the documentation of these items, potential pitfalls can be identified and either corrected by a design change or noted for future consideration.

An interesting approach to interface design specification is to write the interface portion of the help file first, before the programming begins. Again, this forces a developer to consider any issues with the design and usability before time is spent on implementation. Simply, if a procedure is too complex to be easily described in a help file for the end user, then the interface is probably overly complicated to use.

The main goal of design documentation is to determine what the end product should be. This is a “living document”, which means that it should be revisited regularly and, if necessary, changed to reflect any new direction. The design document can help keep the project on track and avoid feature creep or inconsistent functionality. Ultimately, it should be the basis of your software testing plan.

[continued in Getting started]

[This article was originally published in the November 2002 issue of ASPects.]

It can be known as Quality Assurance (QA), Quality Control (QC), ISO9000, or simply “testing”. Regardless of the name used and the particulars of the implementation, it is essential to have a process for making sure that your products exhibit the highest possible degree of quality.

In my previous article, I mentioned the need for testing three times in the same sentence; such is the importance of this topic. The creation of a quality product begins and ends with testing, so it is worthwhile to discuss the different types of testing that can be incorporated into the development process. Although my focus is on development of computer software, much of the process applies to services and other kinds of projects.

The quality process can range from having a fully documented and certified procedure for assuring quality, complete with personnel devoted specifically to maintenance of the process and rigorous testing of the product, down to the lone developer who compiles an executable, runs it once, and then distributes it. It is easy to tend towards the lower end of this scale, through either ignorance or laziness. As independent developers with limited resources, it is probably unrealistic to be at the absolute top of the scale, either. However, we must strive to find a comfort level as close to the top as possible.

When I wrote my first line of code, more than two decades ago, nobody told me about quality, nor even about bugs. The only test was whether the program did what I wanted when the user did what I had anticipated. This changed rather quickly, though, when I got to sit in a computer lab and watch high school students deliberately try to break our games in order to ridicule our capabilities. The challenge became finding ways to make our programs more resilient to these attacks. Though the motivations may be different, today our software is similarly used and abused by our potential customers.

The gauntlet has been thrown down.

[continued in Planning for Quality]

[continued from What Quality is Not]

Keep Looking Up

It is rare that anyone would deliberately ship a shoddy product or knowingly provide a poor service. However, this does happen for a number of reasons. Some of these include limited funding, lack of time, and insufficient talent, which cannot be remedied directly by attitude. On the other hand, sometimes the problem is with perception and awareness, or just a failure to pay attention to detail.

I make the assumption that, as [software developers] who have read this far, you are interested in improving product quality and, therefore, probably already have products that are fairly high on that scale. Nevertheless, we can all improve, and it is important to take the attitude that even the little things make a difference. We need to listen to our users and reviewers closely and resist the natural temptation to dismiss or rationalize criticism. All comments are valid, whether or not they are accurate, and we must pay attention as they tell us where we can yet improve.
Simply by taking the attitude that quality is important and that knowingly providing less than our best for our customers is unacceptable, we can improve both our products and our bottom lines.

Gregg Seelhoff is an independent game developer who serve[d] on the ASP Board of Directors. He can be reached at seelhoff@sophsoft.com.

[end of article]

[continued from Crafting Excellence]

What Quality is Not

In more than two decades in the computer software industry, my experience is that most computer programmers are, in fact, human. For the sake of argument, I will make the logical leap that this also applies to most other jobs within this and other industries. As humans, we are prone to make mistakes occasionally.

Knowing that mistakes are likely to happen, it is important to point out that quality is not the same as perfection, nor should it be. There are too many subjective attributes of even the most basic software to ever achieve perfection, and never shipping any product in the futile search for perfection certainly does not denote quality.

Ironically, mistakes can be opportunities for improving the perception of quality. Most software users are at least sophisticated enough to understand software that does anything worthwhile is bound to have a few bugs. If a publisher or developer is responsive to a bug report from the user, that can potentially instill a greater feeling of comfort, despite the occasional problem. Likewise, involving testers and others in the process helps to create a feeling of proprietorship and, to some degree, loyalty.

Another potential development pitfall that should not be confused with quality is a situation known as “feature creep”. This is the process, basically, in which one begins to think of the omission of a feature as a failure and, hence, continues to add new features and code, seemingly forever. It is tempting to want to incorporate every single feature that any competing product has, but that can be detrimental to the final product. A title with six major features implemented well is more likely to be considered a quality product than one with a dozen poorly implemented features.

[continued in Keep Looking Up]

[continued from Application to Shareware]

Crafting Excellence

Back in medieval times, guilds played a major role in European society. Craft guilds served to train new craft workers and pass the knowledge of an industry down from each generation to the next. A boy would begin training as an apprentice at a young age, learning the rote skills of the craft. After many years of apprenticeship, the young man would be allowed to take a test to prove his knowledge to the guild. If he passed the test, he would become a journeyman.

As a journeyman, the young man would work for a number of different masters, learning and enhancing his fine craft skills. During this time, he would begin creating one very special item to prove his abilities. This would be an arduous task, since the work had to be done in his limited spare time, and the process could take years to accomplish. When finished, this ultimate example of his craftsmanship would be presented to the guild, which could convey upon him the status of master craftsman, if the item showed sufficient craft. This was his “Masterpiece”.

Our company adopted a motto several years ago: “Nothing short of a Masterpiece.” To us, this motto is far more than a marketing slogan. It is a guiding principle, based on the ideas in this article. It represents our joint desire to do our best work on behalf of the company and our clients, and it has become imbued with meaning from shared experiences of both triumph and failure.

In practical terms, the motto says in five small words what otherwise would take much longer to say, if it could be fully expressed. When we have to make a decision regarding an issue that could affect the quality of our work, it only takes one of us to repeat the motto, and we are reminded of our goals and principles. This is certainly not a unique idea; Ford managed it in still fewer words with “Quality is Job 1.”

[continued in What Quality is Not]

[continued from Quality – What is it?]

Application to Shareware

With a working definition of quality, we can attempt to apply this concept to shareware. In this case, the viewer is primarily our potential customer, but it also refers to the press, our possible detractors, and anyone else who can be influenced positively or negatively by any interaction with us, the creators and operators.

The first thing to realize here is that the perception of quality is not limited to our software or services. It encompasses every interaction we have, whether or not we are aware of it. This means our web site, our documentation, our press coverage, and our conversations on unrelated public newsgroups. It even extends to items we cannot directly affect, such as word of mouth, so it is important to convey quality in the arenas that we can control.

It is definitely important for developers to have a quality product, and just the basic issues related to that are a whole separate article. In general, though, this is accomplished by having the fundamental skills, testing, using good programming practices, testing, utilizing available tools and resources, and conducting more testing. Without a good product, the rest of the discussion of quality is basically moot.

Having a quality product, however, is just the beginning. All venues for communication must have similar standards. The documentation should be clear, easy to use, and fully proofread. The product web site should be informative and not appear to be an afterthought. If a phone number is published, that phone should be answered professionally. The ordering process must be straightforward and foolproof.

The pursuit of quality is a mindset that one can choose to adopt by simply refusing to be satisfied with work that is substandard. This mindset should encompass all of the above, plus any other items that could adversely impact the perception of either the product or of the company as a whole.

[continued in Crafting Excellence]