Admin Alert: What’s The Danger Zone For IBM i Disk Utilization?

Working on IBM i and its predecessor machines for 30 years, I sometimes have to modify or redo advice that I’ve relied on for years. Case in point: how full does your disk space have to be before you put your IBM i partition in danger?

I had an experience this month that made me question my assumptions on when you should worry about and how you should set your disk utilization thresholds. Here’s what happened.

What Went Before

In the past, I’ve recommended setting your disk utilization thresholds so that the IBM OS issues this system operator message when disk storage utilization passes 85 to 90 percent.

CPF0907 - Serious storage condition may exist. Press HELP

IBM i automatically sends this message when your disk utilization storage passes your predefined threshold. By default, IBM sets your storage threshold to 90 percent utilization but you can set it lower or higher based on your system needs. I’ve recommended setting this level at 85 percent because that would give you more time to react if there’s a runaway job filling up disk storage. Check out this article on protecting your system from critical disk errors for information on how to change your disk storage threshold.

But That’s Not All

In addition to providing a disk storage threshold for each of your auxiliary storage pools (ASPs), IBM i provides more auxiliary storage system values. These values create a second set of threshold settings, along with an action that will be taken when available system storage falls below your preset value. These system values are:

QSTGLOWLMT–Auxiliary storage lower limit, set to 5 percent by default, this system value provides a secondary threshold of how low available storage (not used storage as with the first threshold) can go before the system takes action. IBM’s default value for QSTGLOWLMT is 5 percent available system storage, although I’d recommend setting it to 10 percent to, again, give you and your system time to react when available storage gets too low.

QSTGLOWACN–The action to take when available system storage falls below the threshold specified in the QSTGLOWLMT system value. You can tell the system to send messages to the system operator message queue; send a message to a specified user for handling storage problems; call an exit point program; put the system in restricted state; or automatically power down the system.

You can find instructions on setting these values in the protecting your system from critical disk errors article I mentioned earlier.

Two Ways For Detecting Low Storage

With these storage values, you have two ways of alerting users or taking action when storage usage breaches a critical threshold: one technique that sends out a message when used storage exceeds a specific threshold; and a second technique for alerting or taking action when available storage falls below a different threshold value.

To reiterate, IBM sets these threshold values at 90 percent for storage utilization and 5 percent for available storage. My understanding is that you must be mindful when passing these barriers because if IBM i storage fills up, it can crash your machine or possibly corrupt your disk and make it unusable.

Testing The Limits

Even though I’ve always believed these values to be solid at 90 percent disk usage and 10 percent free space, it’s unclear how high you can push storage usage before the system crashes. I found that out recently when one of my clients was making a major database change and storage usage on both their production machine and their high availability backup machine unexpectedly spiked well above 90 percent.

We watched nervously as storage usage broached 91, 92, 93 percent and kept going, wondering if the DB change was going to complete before it crashed the machine. Finally, when storage hit 95 percent on the production partition and 96 percent on the HA partition, we called it off and cleared off some additional storage space before continuing the database change.

All of which leads back to my original question: How high can IBM i disk usage go before it causes system problems? In my recent experience, I’ve seen system storage go above 96 percent used and below 4 percent free and the system continued to function. And the fact that IBM sets the QSTGLOWLMT lower value at 5 percent makes me think that Big Blue believes that IBM i storage can run above 95 percent (which backs up my 96 percent disk utilization experience).

Given this experience, my takeaways and modified advice on IBM i storage utilization are two-fold:

• IBM storage is probably more resilient than it was in previous years. It has the ability to run at over 90 percent utilization and for many systems, it can probably run as high as 95 percent utilization. But your risk of a disk or partition collapse increases once you get over 90 percent utilization and IMHO, you shouldn’t tempt fate. Once you get over 90 percent used, do whatever you can to get storage utilization back down in the 80 percent range, preferably lower than 85 percent.
• Again, IMHO the best setting for the disk utilization threshold is 85% full. The best setting for the auxiliary storage lower limit is 10 percent.

If the limits are set this way and you have IBM i monitoring software that sends out alerts when these situations occur, you and your staff will have time to react and bring down storage utilization before it causes system problems.

Joe Hertvik is an IBM i subject matter expert (SME) and the owner of Hertvik Business Services, a content strategy company that provides white papers, case studies, blogging, and social media services for B2B software companies. Joe also runs a data center and help desk for several companies. Joe has written the Admin Alert column for IT Jungle since 2002.

RELATED STORY

Admin Alert: Protecting Your System from Critical Storage Errors

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot