6 Pillars Of Software Testing
𑇐 Manual Testing
Tatiana Timonina
July 18th, 2024 𑇐 12 min read
In this post:
# Pillar 1: Errors, Bugs, and Failures
# Pillar 2: Test levels
# Pillar 3: The role of testing in the Software Development Life Cycle (SDLC)
# Pillar 4: Types of Testing
# Pillar 5: Static testing techniques
# Pillar 6: Dynamic testing techniques
Introduction
In the software development lifecycle (SDLC), ensuring the quality and reliability of applications is crucial for delivering robust, user-friendly products. Testing is an integral part of this process, serving as the safeguard that catches errors, bugs, and failures before they can impact users. Understanding the key points of testing within the SDLC is essential for anyone involved in software development, from developers to QA engineers and project managers. Effective testing not only helps prevent bugs but also saves time, money, and resources, ultimately leading to a more successful product launch and a better user experience.
It is critical to grasp and incorporate the six fundamental pillars of testing to unlock the full benefits of testing. This article thoroughly investigates these essential pillars, which are vital for attaining all the benefits of testing.
Pillar 1: Errors, Bugs, and Failures
- A software error is a flaw in the code. It can occur due to a developer’s coding error where the developer misunderstood the requirement or a requirement that needed to be defined correctly.
- A bug is a code defect. When the application is not working as per the requirement, it is referred to as a defect. It is characterized as a deviation from the actual and expected result of the application or software.
- A software failure occurs when the system fails to perform its intended functions.
Real-world Software Failures and Consequences
The Offender Criminal Management Information (OMNI) tracking system in Washington State incorrectly calculated prisoners’ release dates for over 12 years. A persistent glitch led to over 3,200 US prisoners being released before their scheduled date by December 2015.
Let’s examine this real-world example and identify errors, bugs, and failures in it.
Errors in the OMNI system are primarily human mistakes that occurred during the coding or design phase.
1. Misunderstanding Requirements:
– Example: A developer misunderstood the legal requirements for calculating prisoner release dates, leading to incorrect logic in the code.
– Impact: This misunderstanding caused the calculation algorithm to release prisoners early.
2. Incorrect Variable Usage:
– Example: A coding error where the developer used an incorrect variable representing the number of days to be deducted for good behavior, leading to a shorter sentence calculation.
– Impact: This led to an incorrect calculation of release dates for numerous prisoners.
Bugs are specific defects in the code that cause the OMNI system to behave unexpectedly or incorrectly.
1. Miscalculation of Release Dates:
– Example: A bug in the algorithm for calculating release dates led to the system erroneously deducting more days than intended.
– Impact: Over 3,200 prisoners were released before their scheduled dates.
2. Unaddressed Edge Cases:
– Example: The code failed to handle edge cases where multiple laws affected the same prisoner’s sentence calculation.
– Impact: This bug caused miscalculations for prisoners with complex sentencing rules, leading to erroneous early releases.
Failures were instances where the OMNI system did not perform its intended functions, resulting in significant real-world consequences.
1. System-wide Release Date Miscalculations:
– Example: Due to the bugs in the system, the OMNI system consistently calculated incorrect release dates for a significant portion of the prison population over 12 years.
– Impact: The state of Washington faced public safety risks, legal challenges, and financial costs associated with re-incarceration and loss of public trust in the justice system.
2. Failure to Detect and Fix Bugs:
– Example: The lack of thorough end-to-end testing meant that the miscalculation bug persisted for over a decade.
– Impact: The prolonged existence of this bug led to thousands of premature prisoner releases, demonstrating a failure in the system’s ability to perform its intended functions accurately.
Pillar 2: Test levels
The second pillar introduces the testing pyramid, which emphasizes a large number of fast, automated unit tests at the bottom, followed by fewer integration tests and the least amount of high-level UI tests. This approach promotes thorough, efficient, and cost-effective testing. While unit tests are typically written by developers, collaboration with QA engineers and test automation engineers can enhance the quality and coverage of tests.
1. Unit testing is the first level of testing where individual units or components of the software are tested in isolation. A unit is the smallest testable part of an application, often a single function or method.
Purpose:
- To verify that each component performs as designed (as expected).
- To catch bugs early in the development process.
Examples:
– Testing a function that calculates the sum of two numbers to ensure it returns the correct result.
– Verifying that a method calculates the correct total price of the items in the cart.
Tools:
– JUnit (Java)
– NUnit (C#)
– Pytest (Python)
2. Integration testing evaluates the interactions between integrated units or components and ensures that combined parts of the application work together correctly.
Purpose:
– To detect interface defects and interaction issues between integrated units.
Examples:
– Testing the interaction between a user authentication service and a database.
– Verifying data flow between a web application and a backend API.
Types:
– Top-Down Integration Testing: Starts from the top of the module hierarchy and progresses downward.
– Bottom-Up Integration Testing: Starts from the bottom of the module hierarchy and progresses upward.
– Sandwich Testing: Combines both top-down and bottom-up approaches.
Tools:
– Postman (API testing)
– SoapUI (Web service testing)
3. System testing tests the complete integrated system to answer the question: “Does it do what it should do?” to verify that the system meets specified requirements.
Purpose:
– Validate that the software system works under the functional requirements.
– To ensure that the system functions correctly as a complete entity.
– To test end-to-end scenarios.
Examples:
– Testing a full e-commerce application workflow, from browsing products to checkout.
– Verifying the overall performance of a software application under load.
Types:
– Functional Testing: Validates the functions of the software against the requirements.
– Non-Functional Testing: Tests aspects like performance, usability, reliability, etc.
Tools:
– Selenium (Automated UI testing)
– JMeter (Performance testing)
4. Acceptance testing is the final level of testing before the software is released to the end-users to verify that the system is ready for delivery.
Purpose:
– To validate the software against business needs and requirements.
– To gain approval from stakeholders and users (the system is user-friendly and simple to use).
Examples:
– User Acceptance Testing (UAT): Involves real users testing the software to ensure it can handle required tasks in real-world scenarios.
– Operational Acceptance Testing (OAT): Verifies operational aspects like backup/restore, disaster recovery, and maintenance tasks.
Tools:
– TestRail (Test case management)
– PractiTest (Test management)
Pillar 3: The role of testing in the Software Development Life Cycle (SDLC)
The Software Development Life Cycle (SDLC) provides a systematic approach to software development, ensuring that each phase is carefully planned and executed. This structured methodology helps manage complexity, minimize risks, and deliver high-quality software that meets user requirements and stands the test of time. Let’s examine each phase of the SDLC and the testing activities that occur in each phase.
Most testing activities occur during the implementation phase. This is when coding happens, and unit testing becomes a key focus. Developers carefully write and execute tests for the individual units or components of the software. This important phase initiates thorough and systematic testing efforts, laying the foundation for subsequent testing phases.
Starting as a manual tester can pave the way for a career in testing automation through learning programming languages. This approach ensures comprehensive testing, leading to higher quality and more reliable software.
Pillar 4: Types of Testing
Software testing plays a pivotal role in the software development lifecycle by ensuring the quality, functionality, and reliability of software applications. It encompasses various types of testing such as smoke testing, sanity testing, regression testing, and retesting, all of which are crucial in maintaining the software’s health and stability. The fourth pillow provides an overview of each testing type, outlining its purpose, process, and significance in the software development process. By understanding the fundamentals of manual testing, this guide serves as a valuable resource for junior testers struggling to classify different types of testing and their purposes.
- Smoke testing is a set of simple tests to make sure that the important features are working properly in the product. It’s like a quick test to see if there are any big problems. This test helps us know if the product is ready for further testing or if there are big issues that need to be fixed first.
- Sanity testing is a type of software testing that aims to quickly evaluate whether the basic functionality of a new software build is working correctly or not. It is usually performed on builds that are in the initial stages of development before the full regression testing is performed. Sanity testing is limited in scope and typically focuses on critical functionality and does not aim to uncover every possible error or bug in the system. It is a quick and lightweight way to ensure that the software is functioning as expected before further testing is conducted.
- Regression testing is a crucial aspect of software engineering that ensures the stability and reliability of a software product. It involves retesting the previously tested functionalities to verify that recent code changes haven’t adversely affected the existing features. By identifying and fixing any regression or unintended bugs, regression testing helps maintain the overall quality of the software. This process is essential for software development teams to deliver consistent and high-quality products to their users.
- Retesting is a procedure where we need to check that particular test cases that are found with some bugs during the execution time. Retesting also occurs when the product is already tested and due to some problems, it needs to be tested again.
Now, we will examine examples of all these types of testing in sleep tracker applications designed to assist users in monitoring and enhancing their sleep patterns. By utilizing the sensors found in modern smartphones and wearables, the app captures various aspects of sleep, such as duration, quality, and cycles. Through the analysis of movements, heart rate, and other pertinent data, the app offers detailed insights into the user’s sleep behavior.
Smoke testing. Examples:
- Verify that the app launches successfully.
- Check that the user can start and stop sleep tracking.
- Ensure the main dashboard displays without errors.
Sanity testing. Examples:
- Verify that a new build can record sleep data correctly.
- Ensure that the sleep summary screen updates with new data.
- Check that critical functions like setting alarms or syncing data work correctly after a minor update.
Regression testing. Examples:
- Re-run tests that verify sleep tracking accuracy after adding a new feature.
- Ensure that previously fixed bugs have not resurfaced.
- Check that the data synchronization with the server remains reliable after making backend changes.
Retesting. Examples:
- Re-test a bug where the app incorrectly calculated sleep duration.
- Verify that the fix for a crashing issue on the dashboard works correctly.
- Check that the previous problem with alarm notifications not triggering has been resolved.
Pillar 5: Static testing techniques
Static Testing, also known as verification testing or non-execution testing, is a type of software testing method that is performed in the early stage of development to avoid errors. It is conducted to identify sources of failures early on, making it easier to fix them.
It involves:
– Early identification of issues
– Analyzing code, documents, and other artifacts
– No code execution
Static testing techniques:
- Code review: This involves examining code to find errors and provide feedback. It is a process to detect and remove errors and defects in supporting documents like software requirements specifications.
- Walkthroughs. Developers and testers discuss code and design to find issues.
- Static code analysis tools that analyze code for common errors and potential issues (SonarQube, ESLint, PyLint, FindBugs, PMD)
Features and benefits of PyLint
PyLint is a static code analysis tool that provides numerous features and benefits. It analyzes your code without executing it, identifying errors, enforcing coding standards, uncovering underlying issues, and offering suggestions for code refactoring.
Main Features:
- Checking the length of each line of code
- Inspection of variable names to check compliance with project coding standards
- Verify that the declared interfaces match their actual implementation
Benefits:
Static code analysis helps in identifying bugs and issues at an early stage, saving time and money for the company.
It also assists developers in enhancing the quality of the code during software development. Additionally, you have the flexibility to write plugins to incorporate specific checks or rules.
Pylint Benefits:
1. Improved Code Quality: Helps maintain high standards of code quality and readability.
2. Early Error Detection: Identifies potential bugs and issues before runtime, saving debugging time.
3. Consistency: Ensures coding style consistency across a team, which is especially useful in collaborative environments.
4. Maintenance: Facilitates easier code maintenance and refactoring by highlighting code smells and inefficiencies.
Pillar 6: Dynamic testing techniques
Dynamic testing is a type of software testing that involves running the software and assessing its behavior as it runs. It is also known as functional testing because it focuses on testing the software’s functionality and how it responds to different inputs and conditions. The main purpose of dynamic testing is to confirm that the software works properly without any faults.
It involves executing the software, identifying issues during software execution, and completing static testing techniques.
Dynamic testing techniques:
- Black-box technique
- White-box technique
- Gray-box technique
Conclusion
In manual testing, it’s essential to understand the six pillars of software testing. These pillars help QA engineers ensure thorough testing by distinguishing between errors, bugs, and failures and understanding different types of testing. Thorough testing ultimately saves money by identifying and addressing issues early in the development cycle. This prevents costly fixes after release, reduces the risk of critical failures, and enhances user satisfaction. Investing in robust testing practices is not only a measure of quality assurance but also a strategic financial decision. Differnt types of testing such as smoke testing, sanity testing, regression testing, and retesting help maintain high standards of software quality and reliability, ensuring successful product releases and sustained customer trust.