Alpha testing is critical to find any errors and flaws you may have missed in a program’s development. This post explains how we went about testing My.ComMetrics.com to ensure users’ satisfaction with the final product.

There are three types of land mines in software engineering that must be dealt with when developing software of any kind:

1) Defects: these cause the software to behave in unpredictable ways. Examples could be code showing up when it should not be, failure to handle mistakes properly or incorrectly completed calculations that inadvertently provide the user with inaccurate results.
2) Errors: these can be reduced by using quality assurance testing. The software might perform in a way contrary to what your customer expects. For example, the user enters a URL at My.ComMetrics.com, clicks on View Trend and in response the software provides an empty results page or an error as shown in the image above.
3) Flaws: these are generally viewed by the end-user as errors, without technically being an error within the developer’s code, such as presented data being correct but the visual leaves something to be desired (see image further below).

How to manage software development
To resolve this apparent conflict we need to realize that only defects cause future headaches, and therefore they must not be deferred. Hence, if your software fails to process data correctly, the information provided to the user will be useless.  If the program freezes (remember the Windows XP blue screen of death) the defect must be fixed before it causes harmful damage. Of course, Microsoft proves that one can still be successful while releasing software with defects.

If the software reliably serves you an empty result page every time, this is not a defect, but an error. Nevertheless, it can and must be fixed. By contrast, a flaw is more of a design issue than an error but again, the client expects a fix.

Defects are more critical and the risk of their presence should be minimized right from the start. Terence M. Colligan wrote Nine Steps to Delivering Defect-Free Software, all of which are still useful today, even though it is usually impossible to deliver perfect software right from the start.

Finally, some people call mistakes in software bugs, but we prefer to call them errors, because, for one thing, if they blow up they can do long-term damage. This often includes decreasing customer trust and confidence as discussed in the post Bugs versus defects – there are no bugs just land mines.

Where do alpha users come in?
Alpha users are required because the number of potential errors and flaws in most programs is too great to be able to test for all of them in advance. Instead, alpha users discover the land mines by making use of the software in infinite ways, putting the program through its paces and (hopefully) informing the developer of any eventual problems. They are also the ones who usually point out possible flaws or limitations a software tool may have.

The challenge for the developer is to prioritize the feedback provided and fix the most critical issues first, without forgetting all of the rest of the information provided by the alpha users. This makes alpha users the helpful individuals who enable the developer to refine and adjust the software before releasing it to the public for the first time.

Bottom line
Most projects come with costs that are impossible to quantify at the beginning, including defects. However, the danger with defects is that they can cause so many problems as to bring a project to a screeching halt. On the other hand, software errors are not that bad and can be fixed as needed. Eric Ries discusses this very nicely in his post Lean start-ups vs. lean companies.

In most cases, exhaustive testing is impossible because there are too many possibilities to test them all. For My.ComMetrics.com, we focused on minimizing the risk of software errors and finding the unexpected ones with the help of alpha users. In this way, we continue to refine and improve the software as you read, this thanks to your and other alpha users’ help!

Our question to readers and My.ComMetrics.com users is, what are your experiences with bugs versus defects – does how they are handled make a difference to you? Please share your thoughts and insights on this important issue by leaving a comment. Thank you.