IBM Brings Active-Active Mirroring Into Db2 For i Database
April 24, 2019 Timothy Prickett Morgan
As a platform that is approaching 40 years of deployment within enterprises that can’t afford downtime with their mission critical systems – that’s counting the System/38 as well as the AS/400 and its follow-ons as part of the same continuum – it is no surprise at all that IBM midrange systems running RPG and COBOL had some of the most sophisticated – and perhaps the only application-centric – clustering software ever developed.
Concurrent with the launch of IBM i 7.4 this week, Big Blue is rolling out a new kind of database clustering, which is called Db2 Mirror, that is intended for customers who cannot have any risk of downtime whatsoever in their transaction processing systems. Db2 Mirror is a supplemental feature to the Db2 for i database and is only available on Power8 and Power9 systems running IBM i 7.4. It is also a bit different from the high availability clustering tools sold by Maxava, Syncsort (which ate Vision Solutions, which acquired rivals iTera in 2006 and Lakeview Technology in 2007), DataMirror (which IBM bought in 2007 and then sold to Rocket Software in 2012), and Sheild Advanced Solutions, and we are frankly trying to figure out just how different.
What we can tell you is that Db2 Mirror creates what is called an active-active database cluster across two machines. Which is different from the active-passive setups that many enterprise-grade high availability clustering techniques employ. In the former case, a cluster of two machines are available to do work, and sometimes this work can be unique and sometimes the machines are configured in such a way that transactions commit across a pair of machines at something very closely approximating the same time.
This Db2 Mirror setup is a bit different from fault tolerant machines from days gone by from Tandem Computers and Stratus Technologies, but is clearly inspired by it, and it is also inspired by database clustering technologies created by IBM, such as Sysplex and Parallel Sysplex for the System z mainframe (which started back in 1990) as well as Oracle’s Real Application Clusters (or RAC), which was derived from the TruCluster software created by Digital Equipment for its Tru64 Unix platform and is itself based on the VAXCluster software for the venerable VMS operating system and the RDB database for the VAX platform. Some inspiration no doubt comes from IBM’s own Db2 PureScale distributed database, which was a distributed, parallel database extension of IBM’s Db2 database for AIX and Linux, which scaled up to 128 server nodes in a cluster, much further than the 32 mainframes in a Parallel Sysplex and the eight nodes in an Oracle RAC cluster. And let’s not forget that when IBM was having trouble scaling up its hardware with NUMA technology in the mid-1990s, it created DB2 Multisystem to created a shared nothing clustering setup similar in some ways (but certainly not all) to Tandem and Stratus and Oracle Parallel Server, the predecessor to Oracle RAC. IBM’s PowerHA does disk-based replication of data through mirrored storage arrays and high speed networks rather than database replication through remote journaling across a pair of servers, which has been part of the plumbing of OS/400, i5/OS, and IBM i since 1999.
There are many ways to skin high availability clustering at the system level and at the database level, and none of us has time to explore them all, no matter how fascinating it is. Suffice it to say, Db2 Mirror is a new way, which Steve Will, IBM i chief architect, tells us has been under development for the past three years and comes because some key IBM i customers have been pushing for active-active database clustering on the platform to allow them to have continuous availability – or as much as you can expect to have with two machines mirrored at the database level within a single datacenter.
“Db2 Mirror is not a journal-based thing,” Will explains. “In Db2 Mirror, when you pair two systems together, every database operation happens to the databases on both systems at exactly the same time. So if you do a table update and your application happens to be pointing at the database, that database operation does not complete that update until it has gotten to both systems. This is not a physical replication solution like with PowerHA. A similar thing would happen with PowerHA, but you would have to put that database into an independent auxiliary storage pool (iASP) that would have to replicate to the other side. But then on the other side, that database wasn’t available for doing active work against it. So there is a difference here with Db2 Mirror. We do not depend on journals. You can have independent ASPs, that’s fine, but we do not depend on independent ASPs. You don’t have to access the database in any particular way, old or new style or whatever.”
Given how much active-active clustering goes on out there in the world – you can get it for Windows Server and Linux systems running most modern relational databases, and there are some truly clever parallel, distributed databases inspired by work done by hyperscalers such as Google, Facebook, Microsoft, and Amazon that are available as open source as well – you might have thought that the Db2 for i database at the heart of the IBM i platform could have active-active clustering long since. But there was one real stickler in the way: the single-level storage architecture of the System/38, the AS/400, and their follow-ons.
With single level storage, the operating system treats the capacity of main memory, disk, and now flash storage as a single continuum of addressable memory space and abstracts the differences away as far as the database is concerned. This architecture is truly genius and unique in the world, and thus far only IBM has been able to make it work, and only in its midrange server line. This puts overhead on the system, but it has radically simplified programming, which is the hallmark of the AS/400 and its progeny. But single level storage gets in the way a bit with active-active mirroring, because it is like asking two different people share the same memories and not going insane.
Unix, Windows Server, and Linux active-active clustering technologies have a slightly easier time, according to Will.
“Part of the reason is the how they cluster and then scale. These technologies actually turn over the file system to the storage devices more than that we are able to. One of the reasons, architecturally, that IBM i has not had active-active before this is that we wanted to be able to preserve the single level store architecture but we couldn’t spread it across multiple systems. We had to find a way to do this, where each system still had its own address space because that single level store is how we do our file system. Nobody else has a file system like that. So they don’t have to worry about that. instead they have hierarchical storage systems, they have to spend time doing locking, and all sorts of other stuff that we don’t. There’s nothing wrong with that – it’s just that’s their architecture. But we needed at the database layer was to coordinate every operation so that every operation could be could be mirrored across the cluster. And the more nodes you had, the less likely it would be that we would be able to meet the performance objectives of many of our large clients.”
And that is why Db2 Mirror is restricted to a pair of systems, at least in its initial release. Which is fine. Customers who need more processing oomph can buy systems with more sockets and shared memory clustering using NUMA techniques at the hardware level. And those who need more than that can use Db2 Multisystem to partition their database tables and run them across a cluster of up 32 machines running in parallel in a shared-nothing setup similar to those Tandem and Stratus machines. Db2 Mirror is not about scale, but continuous availability, but it is fun to contemplate a massive IBM i cluster with up to 32 nodes running Db2 Multisystem to scale up the performance of machines by scaling out the hardware and then using Db2 Mirror to have another set of 32 nodes that makes it continuously available. The mind reels. . . .
Db2 Mirror requires the new IBM i 7.4 release, which we report on elsewhere in this issue, but customers are not required to buy it or install it any more than they have been required to buy Db2 Multisystem for more than a decade. Like Db2 Multisystem, Db2 Mirror is integrated into the platform and the database, but it is a separately licensed product – in this case, costing $20,000 per core. Both IBM i 7.4 and Db2 Mirror will be available on June 21, which is the AS/400’s 31st birthday.
We will be drilling down into the architecture and getting some feedback on Db2 Mirror from the folks selling HA software based on remote journaling techniques. What we can say right now is that Db2 Mirror requires a high speed Ethernet adapter linking the two machines that supports the RDMA over Converged Ethernet (RoCE) protocol, which is a clone of the Remote Direct Memory Access techniques built into InfiniBand switching two decades ago. This allows machines to access each other’s memories remotely, but it does have a distance limit of about 200 meters (we heard 1 kilometer as well, but the manual says differently from what we were told). What this means is that Db2 Mirror might be good for continuous availability within a datacenter, but it doesn’t do jack for disaster recovery across a remote datacenter, be it one running in the cloud by a service provider or one operated by a company itself.
RELATED STORIES
The Cognitive Systems/500 2018 Edition
A Hypothetical Future IBM i System
PureApplication Systems Get Power7+, But Not IBM i
Big Blue Pits PureData Appliance Against Ellison’s Exadata
IBM Preps Big Data, Cloud “Project Sparta” Announcements
Oracle Has Built A Modern, Cloudy AS/400
DB2 for i: The Beating Heart of the IBM i Platform
Oracle-Sun Exadata V2, Meet iDatabase V1
HP and Oracle Launch Database Machine, and So Can IBM with i
The System iWant, 2007 Edition
Forget Oracle 10g. Let’s Talk About i5/OS V5g
Tim
According to the manuals which are now available online in the Knowledge Center
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzahg/welcome.htm
The distance between the 2 systems is set to 200 meters maximum, not 1 Km as you have indicated (seems to be a lot of confusion as I have seen 100 meters mentioned on some posts as well? Here is a paste of the content.
On the left are the continuously available Db2 Mirror nodes. These need to be in close proximity as the maximum distance between the POWER servers is 200 meters.
I did ask in a previous ISV council meeting about contention resolution (where 2 nodes have updates for the same DB record) but the information was a little thin about how that would be handled, I seem to remember that it would be at the application level where the contention resolution would have to be managed, but I could be wrong there.
This is one of the items that we have all been trying to resolve as logical replication vendors so that we could get to the goal of Active/Active replication, the availability of Synchronous Remote Journalling did get us part of the way because we could possibly do some kind of record locking at the secondary system to stop conflicts but the chances of conflicts still occur. DB2 Mirror seems to have the advantage because it is not adding the storage of the RJ data and than expecting a separate program to do the lock activity, its more of a continuous flow where the process that sends the data can inspect or hold the target data element? I also remember in the back of my old brain that IBM has some CRC data which would allow remote checking of the DB that is used for other things with the apply journal change but again could be wrong?
DB2 mirror adds a lot to the mix, but in my mind does not affect our Logical Replication products in the SMB arena. The cost of the product $20,000 per core, plus the RoCE card will soon add up to significant costs for those who need to use it, plus you still have to consider DR and replication outside of the Data Center if you need it (I assume you would if you have need for this kind of technology?).
Also would like to add Shield Advanced Solutions HA4i to your list of Logical Replication Solutions.
Chris…
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/db2mi/db2mintro.htm
The design of the high availability system makes me wonder about the tight, highly coupled integration of the OS & DBMS on the IBM i. In the example in the article, the system owner’s intent is to have high availability for the database but, he is required to buy a system that does all the other things the IBM I OS does & this means he incurs an expense for functionality he does not require.
If the OS (and the jobs subsystems that run applications) could be separated from the DBMS and each deployed separately then owners could develop the HA system that they want, not what is forced on them by the unnecessary tight coupling of OS & DBMS. I think that someday we’ll see the IBM i DB decoupled from the OS.