Thoughts on the Coexistence of Full Test Automation and Manual Testing
April 3, 2007 Colin Armitage
|
Automated software testing solutions have been around for more than 20 years but successful implementations are few and far between. Most organizations continue to execute the majority of their testing manually. Industry analysts peg market proliferation at just five percent, with 30 percent being the most extreme estimate. One reason for this damning failure is that users were promised that test automation could address 100 percent of their quality assurance requirements, but bitter experience has shown that total success was far more elusive. For these reasons, it can be said that anyone who tells you that software testing can be totally automated is lying. Expectations are a big part of the problem. The reality is that full automation is not always appropriate and the amount of effort necessary to achieve full automation should never be underestimated. There will always be a trade off between developing a fully automated process and just getting by with the manual testing. The key metric in the discussion of whether it is wiser to automate or not automate is re-use. It will always take longer to develop an automated test bench but there can be overwhelming payback if tests can be executed multiple times. If this “economy of scale” business case cannot be made, then organizations will always logically choose to execute their testing manually. In real life, the situation is very complicated since, for example, any third party application that has undergone extensive modification could be a moving target for testers. With new releases and bug fixes always in the queue, it’s nearly impossible to nail down a stable version to test. If an automated solution cannot adapt to a forever-changing application, then the potential for re-use simply disappears, along with the justification and user enthusiasm for automation. On the other hand, the great thing about automation is that it enables types of testing that are practically impossible to approach manually. Simulating extreme workloads and logging a program’s response can help you avoid large-scale problems once your application goes live. To attempt to test the breaking point of an application manually, dozens of developers or users would be needed for weeks or months and this is simply not practical. Regression testing is the process of comparing the results of initial baseline tests with those of current tests. The object of regression testing is to ensure that areas of an application are unchanged by a new release and continue to work as intended. Full regression testing can provide vital business stability throughout application changes but is incredibly difficult to perform manually due to the time, resource, and knowledge constraints. Life would be so much easier if the barriers to develop automated process could be lowered. All the traditional solutions require in depth programming skills that are rarely found among the quality assurance (QA) fraternity, whose strongest members come from the business side and are subject matter experts, rather than technical gurus. Even if testers can be persuaded to embrace such skills, it can be difficult to find time for the development of test scripts, test the scripts themselves, and train legions of testers in their use, while continuing to make progress with on-going test projects. With seemingly no clear or simple way to transition from a manual testing protocol to one that avails itself to the benefits of automation, people grow confused, lower their heads, and continue with the old way of doing things. In doing so, they lose out on automation’s important benefits: Principally the ability to address one of modern business’ key challenges, which is how do I reduce application time to market and thus deliver real world competitive advantage? You can solve most problems by first adopting a good strategy. To gain the commitment from the business and test teams to improve QA it is essential that the potential benefits from automation are aligned to effort and time expenditure, and can be incrementally delivered when the business requires. I call this approach Crawl, Walk Run. Crawl We all know the importance of an effective plan that enables you to take control and manage the whole testing process. The ability to manage and report progress against that plan is becoming a business imperative due to the ever increasing compliance and audit pressures modern businesses are facing today. As already discussed, a significant portion of testing is and always will be manual, so let’s start there. Hopefully you are already recording your manual testing along with any defects that are discovered. The challenges you face include reproducing defects, constructing test audit trails and verifying the results at visual and database levels. The last six months have seen some exciting developments in this area, not least the announcement of assisted manual testing solutions that can automatically document and track your steps while testing in a non-intrusive fashion, allowing you to correct defects much quicker, by re-creating the problem at a touch of a button. With the initial successes and quick ROI enjoyed in this Crawl phase, you will have more time to continue process improvements which will in turn allow you to benefit even more. Walk Once the testing fundamentals of planning and manual testing have been addressed you should be in a position to move to the Walk phase, which for many companies revolves around the subject of managing test data. What does managing test data involve? In a nutshell it is the creation, management, and re-use of data in a secure and protected environment. Test data management is a cornerstone to successful testing and is an area where technology can often delivery a quick payback. Effective test data creation provides more practical test verification and it also addresses issues relating to disk space, data verification, data confidentiality and protracted test cycles. Companies that do not manage their test data properly will always find any testing automation very painful, and verification almost impossible. Run With the fundamentals in place and a strong hold on your test data, you are in a powerful position to implement full test automation in every area it can be justified. We have already seen that effectively deployed software automation solutions can have a significantly positive effect on functional and regression testing timescales. Implemented correctly, automation can offer great benefits in areas such as time to market, risk reduction, cost savings, and resource efficiencies. Colin Armitage is CEO at Original Software, a company that specializes in the development of automated software testing solutions. Armitage is a renowned expert on software quality control.
|
