Getting Progressive About Regression Testing

October 22, 2018 Jeff Tickner

(Sponsored Content) If you want to employ modern software development and testing techniques, you have to move on from simple unit testing by developers and implement regression testing in your quality assurance (QA) organization. This is perhaps the best way to take the risk out of continuous development – something that companies have to embrace if they are to remain competitive.

The difference between regression testing and normal testing is that in the most common model, the developer has a request to fix a problem or to add a feature, and they make their changes and do unit testing, where they come up with test cases that will test the problem or the feature before they pass it off to QA to essentially run the same tests. Developers make a change and they know what result they are supposed to get back. If you add 2 plus 2, you know you are supposed to get 4. If 2 plus 2 equals four, then unit testing is successful, whereas regression testing is testing all of the functionality, and this is much broader. Unit testing is not looking for broader impacts.

In regression testing, QA tests other use cases that might have been affected by the change that the developer made.

A classic example of the importance of regression testing was something I experienced when I worked at a big retailer a number of years ago. This company was rolling out a new ERP system, and they were using a large number of contract programmers. There was a problem in a sales report, and a contract developer stayed late to fix the problem, pushed the change into production – something the programmer could do because this retailer had no controls – and then went home. Unfortunately, the change that was made also made it impossible to add new sales to the system. Sales just shut down for three hours. They eventually figured out what change was made and fixed it, but the contract developer had just focused on his use case and didn’t realize that the program they had updated was also used by order entry.

The point of regression testing is to test all of the aspects of the business against the changes in the code. There are different ways to manage this.

With mass regression tests, the core functions of your applications are identified, and a scripted process is developed that tests those core functions through QA. This is usually with a batch process and it is usually expensive. But errors are identified and analysis is done to figure out where the impacts originated.

Another regression test methodology, a focused test, involves categorizing your tests, usually by function, and there are tools that do this. So you categorize functions, say those involved in order entry such as adding a new order, changing an existing order, and so on. The test scripts are created to run against these categories.

The third regression test methodology, used by ARCAD-Verifier, is called an optimized test. During the test case you actually record your actions and watch what objects are being touched in the background and put that data into a file. Perhaps you changed three different objects. From a database of test cases, you find and run the test cases that actually touch those three objects. In the case of the retailer, the test case would know that the code relating to the sales report also touches order entry. The tool runs those test cases, too, and would have found the failure and prevented the code change from being propagated into production.

The idea here is to automate the regression testing so someone is not trying to micromanage it. In a modern DevOps world, you want to automate this as part of the code promotion process, so it is not forgotten. Someone is looking at the results of the testing and reacting to it. This continuous testing all meshes with continuous integration and continuous deployment, and every time you toss code to QA, you want to do one of these three forms of regression testing.

What is not well understood about regression testing is that it is about quality, not quantity. And this is about optimizing your testing. What some people don’t understand is that if they have a QA department, they can’t just buy a tool and replace the people in that department. QA people tend to be behind already, and they certainly can’t fire folks just because they have bought a tool. In fact, setting up proper regression testing will require more work, not less work, in the short term, and continuously updating test cases as the data under test changes to reflect the changing applications.

If you don’t put test cases in a database, then every time developers want to test code, they have to develop new test cases. The first step in automation is looking at how the programming and testing resources are being utilized and finding the places where you are repeating the same task over and over again manually instead of providing some way of reusing the effort. Automated regression testing is about reusing those test scripts. The ultimate goal is that you can just push a button to run unattended tests, but the reality is that building a library of test cases, no matter how you do it, takes an incredible amount of effort.

What people do not realize is that when implementing a testing tool, it takes a long time before you see any productivity gains. A long time. The curve is steep. First, you have to build all of the test scripts, and depending on the tool, that could be more time than you are spending now. Then, someone has to maintain those scripts. If you add a new field to a screen and you have scripts in the library that use that screen, those tests are going to fail because in the original tests, that screen did not have that field. It is going to fail every time the test is run, and it is going to be a false negative until someone re-records that test case with the new field in it. There is a maintenance aspect to this, and if you don’t maintain the test cases, they soon become useless. When you add completely new functionality to applications, you have to add new test cases. So regressive testing is an aggressive goal, and it is not aimed at IT shops that are using developers to test right now. Those developers are already way, way behind that curve.

The QA Bottleneck

QA departments are often a bottleneck. Developers do a quick change and it compiles, so that is the first test, and then they do unit testing and it works, so that is the second hurdle. But if the update affects order entry, order update, and order deletion, this is a lot more work. The scope increases with regression testing, and that is precisely the point of it: to cover all of the aspects of the application that could be impacted by the code change.

Ideally, programmers spend about 20 percent of their time testing. In some shops, if the code change compiles they are happy and they just push it to QA or into production. The QA people could be working with a test case that is narrowly focused, and it is really just somebody else doing a second pass of unit testing. The QA team is supposed to be doing a lot more tests than the developers do. With automated regression testing, the QA team goes from heads-down just testing, testing, testing to automating the test cases and looking at the results.

The primary fallacy is that automated regression testing will replace the QA staff. The reality is that your QA team will probably need more people to implement automated regression testing. Your testing footprint is larger, but you catch more errors sooner.

This can be done manually, sort of. IBM’s Rational Developer for i (RDi) has a feature called Code Coverage. When you start up the debugger during application development, it scans the code being debugged for all of the pieces of the application that it touches. You can take the output from Code Coverage and pull it into a database. Then, manage your regression testing by building up this intelligence manually. ARCAD-Verifier automates this manual process completely. The QA staff does not have to be programmers, they don’t have to understand exactly what is happening behind the scenes.

That leads to the next issue: How technical is the QA staff? Typically, the QA staff is not nearly as technical as the developer staff. In a sense, the QA staff is the human equivalent of a regression testing tool: They have a test case, they have some data, and they watch the screen. Some testing tools work like this, and others require you to know quite a bit about what is going on behind the scenes. Creating test scripts does not have to take a lot of skill, but it may take more skill to maintain them. You could just record the script all over again after the application has been changed, of course, but with some tools you can also go in and tweak it.

The key thing is this: If you don’t maintain the library of test cases, it becomes useless over time. Some customers have focused on capturing the test scripts, but didn’t feel maintaining them was as important because it was extra work for the QA folks. Eventually, every time they did regression testing, scripts failed. And because they were expecting those (false) failures, they might miss other, important failures. Even when you are fully automated, you still have recording, maintenance, and interpretation to do, and that takes people. You have to drill down and see why the test case fails.

The other big requirement for regression testing – no matter what the tool, homegrown or a third-party tool – is consistent data. This is often far more challenging than people expect it to be. Customers that have a lot of trouble coming up with a dataset for testing and leaving it alone.

All the testing tools, at some level, are expecting consistent results. When I do unit testing, I am telling the testing tool what I put in and what I should get back. With automated regression testing, you are expecting the results this week to be the same as when you ran the tests last week – across the board, whether it is down at the database level or at the application screens. If I put in a new order, I will get the exact same order number back. If I sell four widgets, it always costs this much. If I put in this customer number, it comes back with this address. If it doesn’t, then something is broken. But if you change the data, the testing will also report that as another (false) error.

Companies sometimes have manual testing, where they are changing the data, but regression testing expects it to be the same. It needs a dedicated set of data. ARCAD-Verifier maintains the data such that when a test is being recorded, it monitors the data in some way – usually through journals or triggers on the files themselves – and when the test is complete, there is a rollback so the data is put back to how it was before the test was run. Those who do this manually could come up with their own process.

For a lot of customers, providing the dataset and keeping people out of that dataset is really hard. As soon as they have a new set of test data, and they have a regular QA environment and they are doing all kinds of terrible things and corrupting files, and they hear of this new set of data, they just jump in with the new stuff. And then they go and mess up their regression testing, which is failing all over the place becomes people were in there monkeying around on their own. That testing data, has to be segregated and kept pure for the test scripts. This seems obvious, but one of the big problems can be that some employees who have *ALLOBJ access to the system can do whatever they want.

The second part about data in regression testing is updating that data. Sometimes the testing itself corrupts the data, so there has to be a refresh process. But if you refresh your data in regression testing, you could potentially break all of your scripts. They expect X and suddenly Y is there. This adds another level of maintenance. If you change the dataset, what is required to update the test cases? If you are doing regression testing manually or with a homegrown method, that could be a lot of updating. Even with a regression testing package that is highly automated, companies have to put a lot of work into this.

One of the strengths of the IBM i is that it is always adapting, and that is true of its applications, too. But when it comes to regression testing, the same holds true. Because we are constantly changing the environment, we have to constantly adapt our regression testing as well. The bottom line is that regression testing is hard, but the results are good.

ARCAD  hosted a Webinar covering continuous testing and regression testing in building a DevOps strategy. You can hear it on-demand at this link.