IBM i Performance Secrets Revealed

The IBM i server is unlike any other computer system on the planet, with unique architectural elements like single level storage and TIMI that simplify programming and boost security. But the 64-bit RISC server also has unique performance characteristics, especially compared to its X64 brethren. To ensure long-term success with the platform, it’s critical to maximize performance.

The folks at IBM recently put together a very informative paper that describes some of the unique performance tracking features that are present in every IBM i system. The paper, titled “IBM i on Power – Performance FAQ,” dives into the various components that comprise IBM i server performance. It can be downloaded here (pdf).

Response Time and Throughput

The paper doesn’t take long to answer one compelling question: What is performance? According to IBM, performance is largely composed of two related but different parts: response time and throughput.

Response time is defined as the elapsed time between the end of a transaction inquiry or demand and the beginning of a response to that transaction. In layman’s terms, it’s the length of time it takes the screen to do something after pressing the “enter” button. Throughput, on the other hand, is defined as the measure of the amount of work performed by a computer system over a period of time.

While response and throughput are related, they’re not the same thing. This is particularly important to keep in mind when assessing the performance of an IBM i server, which is a multi-user system designed to run multiple applications at the same time. While the system was designed to handle a large number of simultaneous requests, that high throughput may come at the expense of response times, the IBMers write.

Fig A. The “run wait time signature” depicts components of wait times of IBM i tasks.

On the IBM i server, response time is determined by how quickly individual threads, or tasks, are executed on the processors. If a task isn’t executing, then it’s waiting. There are three types of wait states, according to IBM, including “ready to run,” “idle wait,” and “blocked waits.” IBM can track these different states at a very granular level through the concept of “wait accounting,” which is exclusive to IBM i and “is a very powerful capability for detailed performance analysis,” IBM says in the paper.

Performance analyses often start with the “run-wait time signature,” which is a graphical depiction of the cumulative time the job has spent in those three different states (see Fig. A). However, this is just the start of understanding how well particular tasks are running on the Power Systems server.

Buckets of Performance Variables

According to IBM, the IBM i OS tracks no fewer than 268 unique wait conditions in IBM i 6.1 and 7.1. “This is possible because IBM has complete control over both the licensed internal code and the operating system,” IBM writes. In other words, don’t knock yourself out trying to get this level of detail from your Linux, Windows, or Unix server.

However, keeping track of all those metrics for each task executing on the processor would be overkill. So for the sake of brevity and economy, IBM assigns each wait condition into one of 32 groups, or “buckets.” As tasks execute, a task dispatcher will watch what goes on, and assign what it sees into one of those 32 buckets.

Wait times can go up or down as a result of all sorts of conditions. At a high level, delays are caused by writing data to disk or reading data from disks, waiting for records to become unlocked, or waiting for IBM i’s journaling function to complete, such as for detailed record-level auditing or high availability purposes (see Fig B).

Fig B. More detail on the run wait time signature.

These conditions, in turn, could be caused by any number of other situations, such as disk pool sizes being too small and therefore causing page faults during writes to disk; having too much I/O overhead; use of synchronous I/O processing; or using journaling too much.

To simplify these performance patterns, IBM characterizes transactions into one of three types, including I/O-limited transactions, soft resources-limited transactions, and processor-bound transactions.

Here’s how IBM breaks these transactions down:

• I/O-limited transactions are transactions that consist of a mix of considerable disk wait times. Getting an efficient I/O subsystem in place, such as by buying faster spinning disks, expanding the cache, or buying solid state drives (SSDs) are effective at speeding up I/O-limited transactions.
• Soft resources-limited transactions, on the other hand, are tougher to speed up. According to IBM, these transactions tend to benefit most from improving application efficiency. (Of course, efficiency is probably why you’re running RPG on IBM i in the first place!) IBM recommends using its tools and facilities, such as Collection Services and Job Watcher, to collect the data, while using the Performance Data Investigator component of IBM Navigator for i and the Windows-based IBM iDoctor client to analyze the data.
• Transactions that consume considerable processor cycles and very little I/O or wait times are called processor-bound transactions, according to the paper. Getting faster processors is the best way to overcome performance problems with these types of applications. Other (cheaper) options include improving the processor and memory nodal efficiency or from improved partition placement.

Every IBM i shop will need to do some tuning to get suitable performance out of their box. The CPW (commercial processing workload) rating of a box is a rough measure of how it will run in the real world, but every application is different. If you’re an SAP customer, for example, you’ll want to load up on memory, while Web serving workloads tend to be more I/O bound. Java-based programs provide a whole additional level of fun for performance gurus intent on hunting down the source of waits.

Benchmarks and Capacity Planning

Since every IBM i installation is unique, some IBM i shops develop their own custom benchmarks to gauge how their performance rises and falls with time and different generations of servers and configurations. This isn’t something that should be taken lightly, and that’s why IBM offers custom benchmarking services through its IBM i Performance & Scalability Services Center in at the lab in Rochester, Minnesota.

The venerable IBM midrange server spawns a fierce loyalty that’s unique in the IT industry, which is why many customers have stuck with box for decades on end. As these customers move existing workloads to new generations of systems, they can benefit from system sizing tools, including the IBM Systems Workload Estimator (WLE), which is another critical tool in the performance guru’s toolbox.

Fig C. IBM provides detailed performance metrics in the dashboard component of IBM Navigator for i.

According to IBM, you can use this tool “to size a new system, to size an upgrade to an existing system, or to size a consolidation of several systems.” In addition to simulating various processor, memory, and disk combinations, WLE can generate recommendations based on different types of partitioning and virtual I/O strategies that the customer wants to use. Because WLE recommendations are based on the customer’s actual application data, and not generic benchmarks, it can provide a high degree of accuracy.

There’s one caveat to WLE: It doesn’t take response time into account. Instead, it bases its assumption on the raw processing power available to it, and assumes that the application is multi-threaded and capable of using all the resources in this symmetric multi-processor (SMP) machine. “The WLE recommendation assumes that your system is well-tuned in terms of performance,” the paper says.

Another performance tool worth keeping in mind is the IBM Performance Management for Power Systems (PM for Power Systems). This product includes PM Agents that collect performance data from individual IBM i servers, and upload it to IBM for storage. When the customer wants to embark upon capacity planning exercises, they can access and analyze the data using a Web browser.

PM for Power Systems is often used in conjunction with the IBM Systems Workload Estimator, which can be used for sizing systems in a server consolidation project or for upgrading to systems with logical partitioning (LPAR). The IBM Systems Workload Estimator uses data collected by PM Agents.

Day to Day Monitoring

Once you have your system tuned or setup, you’ll want to monitor the performance on a regular basis. The IBM i OS has several methods of tracking performance metrics, including basic commands like WRKACJOB and WKRSYSACT or their GUI equivalents in System i Navigator and IBM Navigator for i. IBM Navigator for i, which is the newest and best client that you should definitely be using, also includes a handy dandy dashboard for viewing performance metrics (see Fig C).

In IBM i 7.2, IBM made some improvements to how performance data is collected and displayed in IBM Navigator for i. And in the new IBM i version 7.3, it introduced the capability to visualize monitor data, which basically lets you see all the metrics monitored in one panel.

There are many variables that go into performance, which is why some IBMers in Rochester have devoted their careers to the topic, and why some third-party vendors, such as Midrange Performance Group, have made it their singular product focus.

The IBM paper provides a good place not just for beginners to get their bearings on IBM performance topics, while delving into some of the more advanced topics as needed. You can download a PDF version of the 91-page paper at public.dhe.ibm.com/common/ssi/ecm/po/en/pow03102usen/POW03102USEN.PDF.