Greymine Launches PerfScan To Monitor And Manage IBM i Performance
April 17, 2023 Timothy Prickett Morgan
It is not every day that the IBM i market gets a brand new vendor with a shiny new product, and so today we are celebrating the official launch of PerfScan, a new performance management and monitoring tool aimed at the IBM i market that will expand out to cover all of the platforms running on IBM’s Power Systems – that means AIX and Linux – as well as running on adjacent systems in the datacenter – that means Linux and Windows Server and any weird legacy stuff that customers might ask for.
PerfScan is the first product – and a new product at that, written from scratch – that is coming out of Greymine, a relatively new company founded in April 2020 by system and application performance experts who have been delivering performance tuning and management services to IBM i shops for decades. (We have been hinting that a new product launch would be coming for a few weeks now.)
Gary Patterson, one of Greymine’s co-founders, got his start in 1988 on the AS/400 doing RPG programming and system administration for the Kentucky Lottery, and was eventually the on-site project lead at a consultancy called Butler and Curless Associates on a contract big for Boise Cascade Office Products, which eventually bought OfficeMax in 2003 for $1.15 billion. Patterson had contract gigs with Hewlett Packard, Wal-Mart, Bank of America, CEVA Logistics, and a slew of other companies over the decades, and has expertise in making the IBM i and its predecessor platforms play nice with Windows Server, AIX, Linux, and IBM mainframe platforms – or, probably more precisely, the other way around.
Patterson met Michael Brock, Greymine’s second co-founder, while the two of them were working consultant gigs at Bank Of America. Brock was director of IT operations for Hotels.com in the wake of the Dot Com Boom, and moved over to director of application engineering when Expedia acquired it in 2006. Since leaving Expedia in late 2007, Brock has been president of iPerformance Group LLC, a consultancy specializing in performance optimization on the IBM i platform up and down the stack.
Doug Mewmaw, a long-time contributor here at IT Jungle, is Greymine’s co-founder and vice president of performance services, and importantly, cut his technology teeth at Boise Cascade and OfficeMax, where he went from being a customer service presentative to application programmer to lead systems analyst on AS/400 and iSeries systems to technical support manager across IBM i, Unix, and IBM mainframe platforms over a 16-year career at the company. For nearly a decade and a half after that, Mewmaw was director of education and analysis for Midrange Performance Group, which had what were arguably better performance monitoring and management tools than IBM offered on the AS/400, iSeries, System i, and IBM i platform. HelpSystems acquired MPG back in August 2018 to add its Performance Navigator tool – especially its capacity management capability – to its ROBOT systems management and monitoring belt.
“A lot of people who specialize in performance work came up through the application development side of the house,” Patterson tells The Four Hundred, and it is true of all of the initial people behind Greymine and the PerfScan tool that they have created. “As a result, a lot of the work that we have done has been focused on performance troubleshooting and remediation at the application level. And of course, if you do performance, you get into operating systems, you get into hardware, you get into systems software to figure out where things are going wrong.”
Sometimes, Patterson or Brock have been called in to get a second opinion about some advice that IBM’s sales reps or its Global Services behemoth is asking a big IBM i shop is giving, usually with that advice being to buy more hardware. And because they know the systems inside and out, and they know applications, they can give more cost effective advice. “We did one recently for a customer facing a truckload upgrade with a multi-million dollar system,” says Patterson with a laugh. “And we came in and spent less than $50,000 with us and then a half million dollars on some SSD flash and have solved their performance problems and preserved that system for another four or five years.”
The great thing about a consultancy is that it works directly with customers, and that builds unique experience and knowledge over the decades that is valuable. But the problem with a consultancy – and all three founders of Greymine have run system performance consultancies – is that it can only scale with the number of consultants. To have a greater impact, and do some good for the entire IBM i customers base, what Patterson, Brock, and Mewmaw figured was that they needed to create a new and better system performance management tool that encapsulated their decades of knowledge.
That, in a nutshell, is what PerfScan is all about. And frankly, it is about take a tool further than MPG did with Performance Navigator, and putting together a roadmap to have broad and deep coverage across application and system software running across a wide variety of systems.
“MPG more or less owned the third party performance space in the IBM i market for many, many years,” Patterson explains. “And then when they when they decided to sell to HelpSystems, now Fortra, the acquirer did what companies like that do – this is a legacy product and they have done very little to enhance the product, or for that matter, even support the existing product and customers, other than raise prices dramatically. And by doing that, we feel they created an opening in the market.”
IBM has its own performance tools and services, too, but Mewmaw says that these are really designed for single systems or single LPAR monitoring, but the design goal of PerfScan is to aggregate performance data from a wide variety of systems for a variety of purposes. The initial PerfScan 1.0 product can gather real-time and batch performance on IBM i systems, with the graphics front end that can do pretty reports, visualizations, and dashboards. (These can even be white-labeled for business partners who want to use the PerfScan tool and sell it as the foundation of a managed performance service to IBM i shops.) Going forward, future releases of PerfScan will be able to ingest performance data from AIX and Linux partitions running on Power Systems iron, and then will be able to pull in performance data from VMware ESXi hypervisors and their virtual machines or virtual machine instances running on Microsoft Azure or Amazon Web Services as well as from standalone Windows Server and Linux machines in the datacenter. The roadmap also includes a data harvester for NetFlow Analyzer, which is the network performance monitor for switches and routers from Cisco Systems.
One of the most important features of PerfScan is a centralized event database that can do correlations between performance issues and changes in the hardware and software running across the datacenter.
“Our grand vision for the product is that it will collect hardware and software performance data from a variety of endpoints and be able to correlate back to the application layer and track what changed in a customer’s big complex multi-node network so that we can correlate change events with performance problems,” says Patterson. This, of course, is what the hyperscalers and the cloud builders do, but they have the benefit of also controlling their own Linux variations and building their own hardware and systems software. (Maybe it is not a benefit so much as a horror. Who is to say for sure?) “The point is that any performance analyst will be able to access performance from any part of the network or compute and storage infrastructure without needing to engage a specialist in all of these areas. I have been on calls where 30 people from 15 different teams are trying to figure out what is going on with a performance problem, and everyone literally says, “It’s not me.’”
PerfScan is a multi-tiered program just like most of the applications it monitors. The back-end is written in Node.js, a server-side JavaScript engine created by Ryan Dahl, and the front end is crafted in the React, which is a variant of JavaScript created by Meta Platforms (formerly Facebook). The data harvestors for each platform will be different, of course. The little agent that runs on IBM i right now is written in CL but will probably be recrafted in ILE C once more functions are added to it. Greymine did not want to run its performance collection on top of HTTP, so it created its own communications layer based on sockets and C. Importantly, the agents push data out to the PerfScan tool rather than having PerfScan poll them, which means they are firewall friendly from the get-go.
PerfScan is offered under a software as a service (SaaS) model, and this is particularly important for small and medium businesses that do not want to install and manage PerfScan itself. And, adds Patterson, some large customers will want SaaS as well – one early customer has over 100 LPARs on Power Systems scattered across machines around the globe, and doesn’t want to have all of this performance chatter on its internal network. There is also an on-premises version of PerfScan that runs on a virtual appliance. Pricing for the SaaS version of PerfScan is tiered by the number of endpoints under management, and the on premises version is currently being done on a custom basis because there are just too many variables. Greymine expects that managed service providers are going to be very hot on getting a new set of tools on which to base a new set of performance management services – and we agree.
Right now, Patterson, Brock, and Mewmaw are working on a capacity analysis module which will be available in a future release of PerfScan that will help customers project performance and costing for future on premises and cloud systems where they will run their applications. And of course, the Greymine team is always ready, willing, and able to do a custom performance consulting engagement to help fix pesky performance problems.
PerfScan 1.0 is available now, and you can find out more about the product at this link and about Greymine at this link.
RELATED STORIES
Performance Management – A Personal History, And Turning A New Page
Relieving The Anxiety About IBM i Performance
Melding System Monitoring And Capacity Planning
Where Did My Faulting Guidelines Go?
Memory Management: It’s Your Fault, Now Fix It
Using i5/OS Performance Adjuster to Better Manage Memory
SSD Performance: Be Careful Before You Buy
Why Are Systems Programmers Always To Blame?
AS/400s Are From Rochester, RS/6000s Are From Austin