Expanding Fields Is A Bigger Pain In The Neck Than You Think

October 20, 2021 Andrew Ireland

Remember the Y2K problem, when companies had two-digit dates for years in their applications and the millennial turn was coming and they suddenly needed a four-digit date so all the math would work out right comparing the present, future, and past? Well, every time you need to change a field length in your source code – which almost always means making it bigger, not shorter – you have to suffer through a mini Y2K crisis.

Yes, the Y2K crisis did not bring an end to civilization as we knew it, as some were prognosticating and maybe a few were hoping, and that was because there was a massive, worldwide effort to modernize applications or buy ones – which is why the peak years for AS/400 sales was in 1998 and 1999. But the whole effort was costly and very disruptive.

Having learned from the Y2K experience as well as similar issues such as converting to the euro currency in European countries, there is a much more methodical, and automated, way to attack the field expansion problem, which is still a very big deal even if people don’t think about it much until they come up against it.

Outstanding In Their Fields

We are seeing a big uptick in resizing fields on the IBM i, with many of these fields being in 13.3 or 13.2 or 15.2 formats and going back three or even four decades. Since the coronavirus pandemic hit, many companies are doing much more business than they ever have done in their history, so this is driving field size expansion. But even generally speaking what was fine for a field size for revenues at a lot of IBM i shops three or four decades ago may not be big enough for the amount of business that the company is doing today. Or maybe they want to move to three-digit decimals for finer-grained currency exchange calculations. Or in other cases, companies want to expand the keys on their files – such as the number of customer accounts or the numbering scheme on invoices. That customer count code is a particularly tricky one, since it probably appears in 90 percent of your screens and display files and is used throughout your database files.

And that is the problem with something as simple as changing a field size: The impact can be widespread across user screens, application code, and database files. And if you are going to have to search for these fields manually, keep track of the changes made to them manually, and do impact analysis manually, there is a very good chance that you will miss at least one impacted field and that is when the data corruption begins.

Field Of Dreams, Not Coding Nightmares

Companies often do not realize the extend of their application portfolio. The typical IBM i shop has maybe 2,500 to 4,000 programs and several hundred files; the bigger shops at larger and more complex businesses can have as many as 30,000 programs and maybe 7,000 files in which they have to make changes. It is a massive task, and then once you have made the changes, you have to do a massive regression testing job to make sure the changes are all correct and the application still works correctly.

What happens if you miss one changed field? Let’s say you’ve got an eight character customer ID and your new one is ten characters. All of your programs are writing it out at 10 characters, but you missed one and it is writing it out at eight characters. And all of a sudden you have got corrupt data and you have a big, big problem. You probably now have duplicate records, or you are going to have multiple different customers at the same customer ID.

Because field expansion is such a big problem, and can take anywhere from nine months to more than a year for many IBM i shops to handle, ARCAD has created a service based on its own tooling that automates the mapping of field distribution and interconnections within the application stack, be it RPG, COBOL, or CL code, as well as automating the changing of fields, the mapping of application screens, display files, and printer files with the new fields, and the regression testing to make sure all of the tweaked applications are working properly.

We think it is best to deliver this field expansion as a full service. We can do this on the customer development box, or better still in our own labs, but the regression testing and the deployment has to be to their environments, obviously. We do the analysis with ARCAD Observer, we do field expansion with ARCAD TransformerField, all done with our change management and DevOps tools, ARCAD For DevOps, so we know exactly what has been changed. We build a series of regression tests upfront with ARCAD Verifier and we rerun those regression tests once we have made the field change so that you know you are writing the correct data out of the application. And we have a methodology to allow customers to do parallel development so they can continue to tweak applications as we are making these changes on our development systems. We only have a very short code freeze just before we make the field changes live.

One of our customers reduced a 450 man-day development project down to only 80 days, freeing developers to work on value-add projects. ARCAD Transformer Field typically automates between 50 percent to 100 percent of such mass transformation projects, depending on the quality of your code.

The other big thing that we handle is the deployment of the updated code and the adjustments to the database underneath it with DROPS. This is an optional bit – some customers take it, and some don’t.

One of the big challenges is that if a company changes a field on a really big transaction table, you change the field and then you have to copy a map to the new file structure. That can take days and even weeks if you have terabytes of data in the file. We know of a customer that took four days to change a field on their principal transaction file. We can copy that data off and use journaling and a feature we have called While Active Promotion, or WAP, to promote large datasets where there is a large field change.

These changes require the proverbial drive down the road while changing the tire. So what we do is we deploy the new tables in a staging library. Then we start the synchronization between the production table, which is still live and updated, and its new version in the staging library. We are managing journals. That may take three days and sometimes more. So when it is all caught up, probably over a weekend night, we replace the current tables by the new ones, deploy the new version of the programs and we are good to go. We dramatically reduce that downtime window when the system is not available to do transaction.

If you want to find out more about the ins and outs of expanding application and database fields on the IBM i platform, ARCAD is hosting a webinar, called Expanding A Field On IBM i? Complete The Job In A Quarter The Time, on Thursday October 21, 2021, at 12 PM EDT (which is 5 PM BST or 6 PM CET). You can register for this webinar at this link.

Andrew Ireland is Global Alliances Manager and DevSecOps Business Manager at ARCAD Software.

This content is sponsored by ARCAD Software.

Unit Testing Automation Hits Shift Left Instead of Ctrl-Alt-Delete Cash

It’s Time For An Application Healthcheck

The New Economy Presents New Opportunities For IBM i

Creating Web Services APIs Can Be Easy On IBM i

Jenkins Gets Closer IBM i Hooks, Courtesy Of ARCAD

DevOps Transformation: Engage Your IBM i Team

The All-Knowing, Benevolent Dictator Of Code

Software Change Management Has To Change With The DevOps Times

Attention Synon Users: You Can Automate Your Move To RPG Free Form And DevOps

Git Started With GitHub And ARCAD On IBM i