Selecting the Right Testing Approach – Manual vs. Automation Testing

December 5, 2013
admin
- Automation

Software development teams are always looking for an edge to develop software features more quickly while ensuring a higher level of software quality. In order to achieve these goals, quality assurance teams have to decide the methods and approaches while preparing test strategy. QA teams can opt for manual , automated or a hybrid approach. The approach selection largely depends on certain factors like project duration, complexity, underlying technologies, budget, and resources availability etc.

To select the best possible evaluation approach, apart from understanding project specific requirements, we also need to understand the difference between manual and automated methods as well as the benefits and drawbacks of using these approaches in different project scenarios. In this post, we’ll compare different factors that will help the testers in deciding the best evaluation approach.

Manual Testing

Automated Testing

Project Duration & Frequency:

If software is relatively small and has a shorter period of development, then it means that we will be running lesser number of test cases with less frequency. So, we will opt for Manual.

Project Duration & Frequency:

If the application is expected to be in development for longer period of time, then it means that a lot of regression will be involved. In such cases it is better to use automated – as it will not only reduce long term costs but will also reduce time to run the regression tests.

Allows ad-hoc & random:

Manual is a much better option if you want to do ad-hoc and random testing. It is particularly helpful in such scenarios where not much documentation is available and you want to get the test results quickly.

Facilitates Systematic & Data-driven :

Automation testing is more suitable in scenarios where you want to find bugs by running systematic test cases. Particularly when we have to do data-driven or keyword-driven testing, it is best to go for automated approach .

Not Suitable for Compatibility Testing:

Same test manually need to be run separately on different machines and browsers which is time consuming, tedious and tiresome job.

Quick and Economical Compatibility Testing:

Normally automated scripts are written in such a way that they can be run on different browsers, machines and configurations. So, automated helps us by saving our time and efforts while doing compatibility testing.

Short Term Costs are Reduced:

Manual does not involve script writing in initial stages which reduces the initial time and money investment.

Long Term Costs are Reduced:

Automated involves higher initial investment, but once the scripts are written they can be run for countless times, reducing the long term costs.

Not Suitable for Load or Low Level Interface:

It is difficult to do low level interface regression or load testing manually.

Not Suitable for Usability, UI, Accessibility and Localization:

Automated test scripts cannot be used to check visual references like font colors etc. User experience testing is also not possible via scripts.

Almost Everything can be Tested Manually:

Humans are capable of checking anything that is testable. All that matters is the knowledge and skill set of the resources.

Everything cannot be Automated:

There could be plenty of test scenarios and application areas which are either impossible or are very costly to be automated. For example, checking flash based application areas is not easy and effective.

Good for Finding New and Un-expected Defects:

It is because it involves thinking and learning. One can find bugs by using the application in ways it was not intended to be used.

Generally does not Find New and Un-expected Defects:

It is merely the sequence of actions that are scripted, which can only find the expected bugs.

Application Developed on New Technologies & platforms: Automated tools for new technologies and platforms usually take years to develop and mature and that is when manual test fills the gap and fulfills our needs.

Mobile Applications Across Different Devices:

There are hundreds of mobile devices available in market today . It is not possible to manually test the application on all of these devices and that is where automated tests helps us to quickly test the apps on different devices using services like AppThwack or CloudMonkey.

Manual Tests provide Limited Visibility:

If developers, team leads, managers want to verify and see the test results by themselves they have to repeat all steps. Additional effort is required every time.

Automated Tests provide Global Visibility:

Developers, team leads, managers can run the tests with single click and see the results by themselves without any additional effort.

Conclusion:

In light of the above comparison, it can be concluded that both manual and automated have their strengths and weaknesses. None of these approaches can replace each other. It is better to adopt the automated approach for those areas of application, whose requirements will not change frequently and where a lot of regression will be involved. On the other hand it will be better to use the manual approach, when the project is short termed, involves less regression and when we want to do the usability, accessibility and GUI testing which is not possible via automated test scripts. So, we can conclude that final approach towards checking any application can vary from project to project depending on all these factors. It is up to us to identify and analyze all these factors, and decide which areas should be automated and which are better to be tested manually.