In this article we discuss the concept of unleashing excellence in striving for perfection and what it truly means in the context of software testing. While perfection is commonly defined as the process of improving something until it is flawless, here we explore how understanding the 7 ideas listed below relate to the striving for perfection in the field of testing.
- Testing shows the presence of defects, not their absence.
- Exhaustive testing is impossible.
- Early testing saves time and money.
- Defects cluster together.
- Beware of the pesticide paradox.
- Testing is context-dependent.
- Absence-of-errors is a fallacy.
By examining these points, I hope to shed light on how testing can grow into a successful process. And how these points could help inform and shape your own testing methodology and ultimately unleash the potential for high-quality software.
Before getting into the crux of this article, it should be noted that each of these points can be interpreted differently for each person. But it is nevertheless important to understand the concept in order to walk away with a key element from each point.
Let’s get to unleashing.
1. Testing shows the presence of defects, not their absence Testing software is the detail-originated task that reveals the presence of flaws, bugs or defects rather than guaranteeing their absence. In this context, defects refer to any deviation from the intended functionality of the software resulting from errors in code, design or requirements. The primary responsibility of a software tester is to meticulously scrutinize software to uncover these defects, highlight their existence and acknowledge that they do, in fact, exist and as such that the product is not without them. A product will always contain defects, and the only “ZERO-BUG” software currently out there is “ZERO-SOFTWARE”. Zero- bug software merely means that all known bugs in an open state have been addressed. While software testing is an essential process that helps identify and remove defects, it is not possible to test all possible scenarios, conditions and combinations that a user may encounter in real-world usage. As a result, one can think of it like Schrodinger’s cat – there may or may not be defects.
2. Exhaustive testing is impossible As stated above, testing every conceivable scenario, condition and combination that a user may encounter in real-world usage is impossible. Despite a tester’s best efforts to generate a plethora of test data, test users and testing scenarios, the sheer number of possible outcomes makes comprehensive testing an unconquerable task. Testing, by nature, is time consuming. It is bound by testing effort, allocated time, project deliverables, requirements and deadlines. Consequently, exhaustive testing within a specified timeframe presents an added layer of complexity to the testing process. One might think: “But wait !” What about smaller software systems? The number of possible test cases can quickly become too large to be tested and yet again limited by time and/or resources. So where to from here? It is important in testing to identify efforts and focus on the most critical and important scenarios, the “Happy Path”, as some call it. This way, a tester can cover the most vital or most used paths within the system.
3. Early testing saves time and money There is a misconception that testing only starts once a requirement of a product has been delivered, but this is far from the truth. Testing can begin as soon as artefacts are produced or code started. There is a concept in testing called White box testing, which is testing the inner workings of a system, this allows a tester to see how a system fits together and how the logic flows, This gives a tester:
- Insight into how the functionality would perform;
- Direction of their test cases; and
- The option to raise concerns and risks at a much earlier stage in the process.
4. Defects cluster together In short, this is known as defect clustering. This is the observation that defects tend to occur in clusters or groups. This ultimately means that there would be parts of the system that have a disproportionately high number of defects while other parts have very few defects. This usually occurs as some parts of the system have more complex logic and flow and require a lot more development to function as needed. This in itself already shows how defects can cluster according to the above, but it also means that parts of the system that are integrated with said complex module could subsequently also lead to creating a cluster of related and unrelated defects. So how can one circumvent this? Understanding the concept of defect clustering means that as a tester, one can develop testing strategies to identify and address defects using an effective testing strategy along with testing early using static testing strategies. This, in turn, relates back to point 1 of testing, which shows the presence of defects, not their absence.
5. Beware of the pesticide paradox No, we are not talking about actual pesticides which one finds in agriculture but the tasks involved in finding defects. Spraying the same pesticide over and over again can lead to pests developing a resistance making it less effective over time. The same can be applied to testing. Running the same test cases repeatedly without any adaptation or modification over time leads to diminished results in the returning of new defects. Remember, just like in agriculture, software can become resistant to those tests and new defects may go undetected. Just like how we can change pesticides in agriculture, we can adapt our testing approach to avoid this paradox. As a tester, we can avoid the paradox by:
- Modifying the test cases where the test cases can be adapted to cover new scenarios or edge cases and thus discovering new defects.
- Employing Exploratory Testing. Exploratory testing is a testing technique that can find defects that scripted tests can not. This allows the tester to have a more free-form approach allowing testers to identify defects that might not have been anticipated.
- Using a risk-based approach whereby testing efforts are based on risks of modules. This allows the highest-risk areas of the system to receive the most attention while lower-risk areas are still tested but with less emphasis.
6. Testing is context-dependent. Just as everything in life context is needed, testing is no different. There are a lot of various contextual elements that need to be considered when understanding how testing happens within certain contexts. Things like testing approaches, methods and techniques would change or shape the way in which testing is done. These can vary based on the nature of the software. How does context affect the development project and testing? Development methodology has a wide array of options, and depending on the development methodology, the testing approach may be different, i.e. Agile being iterative releases means continuous testing throughout, while DevOps may emphasise a more automated testing approach. The target audience may influence the types of testing happening as well. Software specifically created for an industry or profession may require testers to focus on more specific functionality or usability elements that would be critical for that specific user. By recognising that testing is context dependent, we as testers can develop more effective testing strategies that are tailored to the needs and requirements of the software development project. This, in turn, ensures defects are identified and addressed in a way that maximises the software quality and usability.
7. Absence-of-errors is a fallacy The absence of errors is the binding and endpoint for all of the above-mentioned points. It is based on the recognition that software testing can never guarantee that a software system is entirely defect-free no matter how much testing is done ( “2. Exhaustive testing is impossible”). Yet again, it is impossible to test every input and combination that a system may encounter, and the very nature of software development being iterative means that new defects may be introduced and not anticipated as changes are made (“4. Defects cluster together.”.) While testing is an essential part of ensuring quality (“6. Testing is context dependent.”), it is important to balance efforts and to implement testing as soon as possible (“3. Early testing saves time and money”). This ensures that quality is delivered and reduces the risk of defects going undetected. (“1. Testing shows the presence of defects, not their absence”) Understanding that the “absence-of-errors” is a fallacy can help developers and testers form a more realistic expectation about software quality. Taking the 7 principles of testing into consideration when testing helped us build better software and ensure its quality at Polymorph.
