In The IBM i Trenches With: Chilli IT
May 9, 2022 Timothy Prickett Morgan
There are many strengths in the IBM i platform besides its hardware and software architecture or the sophistication and ease of programming embodied in its development tools. The diversity of its partner ecosystem is another strength.
As we pointed out last March with the launch of our ongoing In The IBM i Trenches series, globally there are thousands of resellers, each serving tens to hundreds to sometimes thousands of customers, as well as thousands of suppliers of third party maintenance, of technical support, of programming and system management services, and now of hosting and cloud computing. We want to get some insight from these companies, many of which are small and often family-owned businesses who have been in the IBM midrange for many decades.
This week, we are talking to Chilli IT, a managed services provider and reseller for Power Systems machinery running IBM i as well as IBM Storwize and FlashSystem storage that was founded in 2003 and that is located in Chester, just a stone’s throw away from both Liverpool and Manchester in the northwest of England, one of the manufacturing and distribution centers of the United Kingdom and therefore a natural home to the AS/400 and its successors.
We sat down with two of Chilli IT’s co-founders, Richard Warren, managing director, and Stan Wilkins, technical director, who have a long history in the IBM midrange dating back to the time that they all worked at Oak Brook International, an offshoot of the European arm of Real Solutions, the software and services arm of giant systems reseller and lessor El Camino Resources based in Southern California back in the 1990s. The Oak Brook name is a reference to the Chicago neighborhood where high availability software supplier Lakeview Technology, the creator of the MIMIX tools, was located, and if you are curious, the name Chilli is a reference to the “hot topic” designation that AS/400 architect Frank Soltis used in his Inside The AS/400 book published in 1996.
Chilli IT has 20 employees and its services business has 50 customers who collectively have hundreds of systems and a total of 240 logical partitions under management, and it has been focused predominately on customers in the United Kingdom, as you might imagine. But given that the world is global and now accustomed to working remotely, the company is looking to expand its reach further into Europe and North America, and we are doing our part to help by letting you get to know the company better.
Timothy Prickett Morgan: Thanks Richard and Stan for joining us. To start, tell us a bit about how and why you started the company.
Richard Warren: Initially, Chilli IT was really a continuation of things that we had been doing for a number of years. So a large part of it was at first professional services around consulting: Solving problems about installing operating systems, doing migrations, all kinds of typical systems engineering stuff. We have never really been involved with application development, although we do consult on development from the performance point of view. It’s always been about the operating system.
TPM: I am happy to know a lot of companies like yours and their founders, and they don’t appear to sleep as much as I do. . . . I presume you do a lot of work with HA, given your history.
Richard Warren: We do high availability as well, which is something we’ve always done.
Going back probably 12 years or so, we had some customers that wanted us to consider looking after their systems. All we had were really experienced guys with lots of skills that were quite expensive and there was no way we could sit them in front of a terminal and ask them to check boring things every day.
And so at that time, we decided that we needed to find a way of delivering managed service in a way that suits us and in a way that we can deploy quickly and at a price point that would suit the customers, which meant an awful lot of automation was going to be needed. That need coincided with Linux getting a level of maturity to the point where Ubuntu was maturing and Red Hat and SUSE Linux were established. . . .
Stan Wilkins: And we were looking at any alternative to having customers buy a lot of Microsoft Windows Server licenses.
Richard Warren: That was the other thing as well. Anything that can help fledgling business save some money is a good thing.
So over the years, we’ve developed our own toolset for managed services, which oddly enough is all based on open source software.
TPM: Talk to me about that toolchain. Did you start with Nagios and Ganglia, or do something else?
Richard Warren: We needed a core piece to start with, which needed to address a couple of things. One, we needed to look at the OS and things in the OS to make sure they were all okay. The other thing we needed to do was we needed to look at how MIMIX and iTera HA software was doing because we were involved with both of those at that time. Other pieces of monitoring and middleware and whatever the tools that are out there tend to be based around the APIs, and while there are some pretty good tools out there, they did not necessarily have the flexibility that we wanted.
So we took snippets of code that open sourced the TN5250 emulator and the 5250 protocol, and we decided the best thing to do would be to code something up that would pretend to be a person and log on to the boxes and navigate around and get the information that we want to pick out from wherever we wanted to get it from. So it could just go and do the hard work for us.
TPM: So you basically created a bot, then.
Richard Warren: I suppose you call it that. So we developed a scripting language of our own, fit to the AS/400 and iSeries and IBM i.
TPM: Wait, you didn’t look at Perl or anything like that, you just sat down and wrote your own scripting language from scratch?
Richard Warren: We just we needed something intermediary, and so the code itself was actually written in Python. We were quite early adopters of Python, which I know is really, really trendy now.
TPM: Did you get lucky in that choice or was that prescient?
Richard Warren: Well, Python is just a lovely language. To be honest, we just looked at the languages that were available at the time and it was a nice clean language and it comes with an awful lot of class libraries that help you with all sorts of different things as well. So for instance, one of the things we wanted us to do was send emails out from an alerting point of view. Well, Python has an SMTP email library, so you can send an email out in like six or seven lines of code.
TPM: Let me ask you a question. If you had to do it all over again today, would you use Go or Python?
Richard Warren: I would probably want to do something in Rust, actually. There’s a little bit right in the middle of our tool that processes an awful lot of information. And it’s still doing what we want it to do with Python, but it could do it a little bit faster.
The basic concept of what we created was a tool that wandered around the OS, that understood all of screens and would chuck it to this piece of code in the middle. And in that middle, there was a set of rules that would automate all of the checks that a human would have had to do, interrogating the software, so we could automate all of the checks. So the actual human beings could just deal with the issues when they popped up.
TPM: Was this actually running on in PASE on OS/400 or IBM i?
Richard Warren: We run the tool on Linux on Intel, and seven years ago we looked at running it on Linux on Power. But unfortunately, we were getting memory leaks. It just wasn’t built that well at the time.
TPM: And not every IBM i shop wants to have a Linux partition to worry about, I suppose. Do you drop an X86 server at the customer or run this tool remotely as part of your services?
Richard Warren: It’s a bit of both. Some customers have security constraints, which means we can’t put it in their infrastructure. All we really need is an IP connectivity to the systems. Sometimes it is put into a virtual machine running in their VM farm. Sometimes it’s on hardware, sometimes it’s actually via comms link. All we really need is to link to Port 23 because all we’re really doing is logging in and getting the information. And then over the years, obviously, we’ve developed that to take in Unix interfaces, VIOS, Storwize, and we make it do stuff with AIX and Linux as well because customers asked for that.
TPM: Do you commercialize it as a piece of software that that other people could use or are you just going to keep this to yourself?
Richard Warren: It’s never been designed in that way. If someone said this is brilliant, can you turn it into a product for us and we’ll guarantee a chunk of money for you to do it, then it’s something we might consider but. . . .
TPM: All you would do is foster your own competition in managed services.
Richard Warren: Essentially. We have been developing the managed services business and we do a lot more with professional services and have taken on board a lot of new products over time. We were really early on external storage, for instance, and seeing the merits of doing a flash copy and spinning up a test LPAR really quickly. Our business remains three pillars: Professional services, managed services, and traditional IBM reselling.
TPM: How do you position yourself against the cloud, against other managed services providers, and against IBM?
Richard Warren: The cloud is an interesting situation, and we have developed a partnership with Skytap.
We don’t have datacenters, and we have absolutely no interest in owning a facility that houses customer equipment. So everything we do is remote. In our view, whether it’s on a customer’s premise, whether it’s in a third party datacenter, whether it’s in the cloud – really doesn’t matter to us. We’re interested in looking after the OS and making sure the environment is healthy, and that the HA software works. We have done migrations between datacenters using MIMIX and live migration.
Stan Wilkins: If you look at what we have been doing in storage, we have been deployed in HyperSwap for the last number of years. We’ve been doing remote mirroring backups, remote mirroring disk as well as a software-based safeguarded copy. Since last October, we have been developing Safeguarded copy, and we realized really quickly that there’s no point doing an immutable copy if you are not going to validate it. In our opinion, it’s a complete waste of time if all you’re going to do is do an immutable copy and hope it’s okay. We think you do an immutable copy and then you validate it and then you know you can use it, you don’t just keep making successive immutable copies and hope one of them works. We have written the code that does that copy and validation, end to end.
TPM: What is your desire? You’ve got a couple hundred partitions, a couple dozen customers, two dozen people –that’s your scale, a good-sized small business that is very, very focused and with deep experts. But clearly, I think you can do much more, and I think this business can scale at this point in the history of IBM i and in IT in general because geography doesn’t matter as much as expertise.
Richard Warren: We’re not relentlessly pursuing growth. Yes, we want to grow. We can take on more customers and more work and automate and grow it in a more organic way than rather than a rapid growth. But part of the problem is that if we wanted to go out and hire more really good IBM i techies tomorrow, I don’t know that we would find them. So to some extent, this is the constraint and that says automation is the way to go and providing the right environment for the team is what we need to do.
Stan Wilkins: With professional services, we are getting involved in a lot of very big corporate migrations, where they’ve obviously got, large links with third party, system integrators but are finding themselves coming back to the very specialist people who can do that IBM i bit. They are not afraid to go to a smaller company because we are experts in that part of the corporate migration. And we’re also doing quite a little bit of that with VMware ESX because we’ve got customers who have gone to external storage and now they want to put their X86 server VMs on this storage. We can show them how to do it, we can hold their hand. Our motto is: Become that trusted partner.
We’re not there to take over their world, but we’re just there to make sure they do it in the right way.
RELATED STORIES
In The IBM i Trenches With: LaserVault
