Software testing professionals have been discussing the concept that testing is classified into two types, which are manual and automated testing for decades. Despite appearing to be clear and simple, this type of framing has emerged as potentially misleading and not very useful to practitioners, failing to promote efficient and accurate assessment of software products. While undertaking this exercise of defining software testing in the current world that is filled with constant change, it is time to let go of the kind of thinking that would separate testing as a discipline and testing tools in a binary manner, as developed fully embodies and delegatory a form of testing.


Importance of Terminology in Quality Assurance

In our field, the terminology we use is critical for both understanding and practice. A foundational principle of Quality Assurance (QA) is to critically evaluate every aspect, including the language we employ. Often, software defects are discovered simply by scrutinizing the terminology used in documentation or discussions.

A Critical Examination of Manual Testing

Defining Key Terms:

  • Manual: A method or approach to executing tasks.
  • Testing: An action that fundamentally engages analytical and critical thinking.
  • Tester: An individual who conducts tests.

As Cem Kaner so perfectly puts it, “When we’re testing, we’re not talking about something you’re doing with your hands, we’re talking about something you do with your head.” The term "manual" when related to testing or testers can diminish the true essence of software testing, reducing it to simple execution of test cases. While "manual" may be appropriate for discussing test execution, it does not capture the scientific essence of testing itself. Critical thinking is not a "manual" process. Testing transcends both manual and automated classifications; it is fundamentally an intellectual endeavor. It is essential to differentiate between testing and the methods employed to execute tests. (Bolton, 2023)

A Critical Examination of Automation Testing

Defining Key Terms:

  • Automation: A method or approach to executing tasks.
  • Testing: An action that fundamentally engages analytical and critical thinking.
  • Tester: An individual who conducts tests.

As Alan Julien aptly notes, "Testing cannot be automated." This statement underscores the distinction between testing—an analytical and critical thinking process aimed at identifying software defects—and test execution, which involves merely running a test. Testing is inherently a thoughtful, intellectual activity and cannot be categorized as either manual or automated. While we cannot automate the critical thinking involved in testing, we can automate test execution. Thus, terminology that accurately reflects the nature of automation in test execution is crucial; a more precise term would be "Test Execution Automation."

In QA, precision in our language is paramount. We must avoid conflating the act of testing with the methods used for test execution. The terms "automated" refer to execution methods, not approaches to testing. It is common to mistakenly associate the entire act of testing with test execution, yet testing encompasses much more than just the tools used for execution. As a result, we may overlook numerous ways in which automation can enhance, expedite, and facilitate various testing tasks. These tasks include data collection, product or test suite setup, product reconfiguration, installation, behavior monitoring, logging, stress testing, fuzzing testing, report parsing, output visualization, data generation, input randomization, automation of non-check repetitive tasks, algorithm preparation, and version comparison. This list is far from exhaustive. You will find that we failed to consider how tools could assist us, neglected to utilize them effectively, and used them without sufficient skill, which ultimately compromised the quality. (Bolton, 2023)

The Flawed Dichotomy: Deconstructing the Binary

In its simplest form, testing is a cognitive activity that aims at assessing a product through usage, trial, and exploration. This includes activities such as gaining knowledge about the product; determining risk; developing and selecting tests; understanding outcomes; and making evaluations about the quality of the product. None of these activities are inherently “manual” or ‘automated”. When people discuss manual testing and automated testing, what they are usually talking about is the tool being used to perform the test. Nevertheless, tools are the only apparatus that may help out in performing specific tasks. They can never do what testing using human input involves, since it involves critical thinking, judgment, creativity, and flexibility to change.


The Role of Tools in Testing: Augmenting, Not Replacing

The general idea is that tools cannot do testing alone but can be tremendously useful during testing. Tools are typically used to run scripted tests, enter data, and perform other tedious and repetitive tasks. They can also gather and process a significant volume of information and give results that are too massive for a person to deal with. However, it is necessary to note that tools themselves cannot replace the professional to some extent. All they are capable of doing is the task assigned to them and they have no way of knowing what the testing or the product is all about or judging the quality of the product.


The Importance of Human Judgment: The Irreplaceable Element

There is always the human factor involved in all the testing exercises. Since it is required to develop tests, analyze test outcomes, and make choices about the quality of the product, it is necessary. People are also able to communicate and change their behavior and may try to look for new ways to test specific products. It is important to forecast that, despite numerous and diverse use cases, tools, in particular cognitive ones, cannot and must not replace human capabilities. Lost in the debate over automation and manual testing are the realized synergies between the two approaches. Tools offer efficiency and speed and heavy evaluation and decision making is done by humans.


Supporting Evidence: Research and Expert Opinions

Well-known specialists in the sphere of software testing provided a revealing analysis of the drawbacks connected with attempts to draw a strict line between manual and automated testing. James Bach, the testing consultant, has been trying to opine that "testing is neither manual nor automated. It's a cognitive process that requires human judgment and creativity.” In the same way, another important figure in the testing community, Cem Kaner, also stresses that “testing is thinking, not doing.”


Moving Forward: A More Nuanced Approach

It is not a question of whether to choose manual working or automated working, instead, there should be certain tasks where a certain working style is applied. This can range from simply applying tools that are used to do a lot of the work automatically to employing exploratory testing to identify these flaws. It is, therefore, necessary to understand what testing truly is, and what properties of humans and tools make for effective testing. We can also define a better organizational structure where humans and tools will be blended together to guarantee quality software products.




The Need to Move Beyond the Terms "Manual" and "Automated"

It's important to recognize that the terms "manual testing" and "automated testing" are not only unhelpful but also potentially misleading. These terms imply a binary choice between two distinct approaches when, in reality, testing is a much more complex and nuanced activity. It is more logical not to think in terms of these terms and instead think about what activities testing encompasses, and then select the best strategy to achieve it. This can include employing tools that will handle the repetitive tasks, doing contextual testing to identify more problems, or both. Getting rid of such a simple division as ' manual and automated’ testing allows us to work towards a better strategy for software evaluation. It will help us develop applications with fewer bugs and glitches and ultimately present our users with a better solution.

Conclusion

The idea of ‘manual’ and ‘automated’ testing forms two poles of a false dichotomy that masks the actual nature of testing. The shift from this stereotype is necessary to open up a new paradigm wherein tasks are reidentified and human and tool are given their rightful places. Thus, it becomes possible to improve the testing process and create more effective software products. The terms "Manual" and "Automated" testing should not be considered as distinct types, methods, or approaches to testing; rather, they represent different methods of test execution. It is common for individuals to conflate the overall process of testing with the execution of tests. However, testing encompasses far more than simply executing tests with or without the aid of tools.

Context is crucial in testing, and a strict balance between manual and automated methods is not only impractical but also not necessarily beneficial. Each testing activity should be analyzed individually, and subsequent decisions regarding the use of tools for test execution should be made based on this analysis. Ultimately, the focus should be on approaches that provide the most logical and effective solutions. It is important to note that testing itself is not inherently "Manual," nor can it be fully automated. (Bolton, 2024)



Last update: 05/11/2024 

References:

Bolton, M. (2024, October 22). On balance. https://developsense.com/blog/2024/10/on-balance

Bolton, M. (2023, September 2). “Manual” and “Automated” testing. https://developsense.com/blog/2013/02/manual-and-automated-testing

Bolton, M. (2022, December 5). “Manual testing”: What’s the problem? https://developsense.com/blog/2021/04/manual-testing-whats-the-problem