Oracle Has Built A Modern, Cloudy AS/400

October 10, 2011 Timothy Prickett Morgan

I just spent a week off and on with Larry Ellison, co-founder and chief executive officer at Oracle. I wasn’t playing against him and tennis buddy and former Hewlett-Packard CEO, Mark Hurd, at the Pacific Athletic club, laughing about how the rest of the world lives. No, I was attending Oracle’s OpenWorld customer conference, which besieged San Francisco last week with 45,000 attendees, and a bunch of announcements that preceded it. Luckily, I was attending these events through YouTube, from the comfort of my very own office.

For the past several weeks ahead of the OpenWorld extravaganza, the top brass at Oracle announced a number of new systems aimed at midrange shops. First up was the Oracle Database Appliance, a cluster of two X86 servers designed to support Oracle’s 11g R2 databases and its Real Application Clusters (RAC), which runs a single database across multiple server nodes. This was followed up by the announcement of the Sparc T4 processor, which crams eight cores and 64 threads into a single chip and which also gives IBM a run for the money in the standard, SMP server part of the midrange.

I’ve been thinking a lot lately about the juxtaposition of Oracle’s “engineered systems” and IBM’s “workload optimized systems,” not just because these things are grabbing headlines, but because all the major system makers are trying to figure out a new way to sell a very old idea: machines designed to do very specific work rather than being general purpose systems. Only this time around, more than a few of them are trying to make these engineered systems out of commodity processors, memory, disk, flash, and networking components. The secret sauce–and the profit–in each one of them is not the hardware, but how a collection of hardware supports a very specific stack of application software.

There are many layers of genius in the original Application System/400 machines that IBM announced back in June 1988, when Oracle was just adding row-level locking to the Oracle V6 database and had put out a transaction processing overlay for it as well as hot backup and the PL/SQL. This was also the year after Oracle started its application division.

We all know about the technology independent machine interface (TIMI) at the heart of the AS/400 and its progeny as well as the integration of the DB2/400 relational database as the file system for the machine. Another genius of the AS/400 design that matters with regard to what Oracle is doing today is the asynchronous processing that was embodied in the AS/400s right from the beginning–and something that IBM has largely abandoned with the Power Systems lines of the past few years. There is no such thing as an AS/400 processor. There were many AS/400 processors, all working asynchronously with each other to do central and I/O processing. You broke the job up so you could have cheaper processors doing the grunt I/O work and not bother the very expensive CISC processor that was running most of the operating system. This approach yielded impressive profits for IBM since it could create a fairly powerful entry and midrange system based on modestly powerful processors rather than a big, expensive CPU like that in its mainframe. The software may have been more difficult to write and tune, as is certainly the case with the code behind Oracle’s clusters today, but the results were more than the sum of their parts.

While IBM was putting a collective of 16-bit and 32-bit Motorola 68000 processors at the bidding of 48-bit CISC processors (which I have always contended were modified 68K chips in the first place) in the original AS/400, all in the same system and connected through a peripheral bus, Oracle has a more modern, clustered approach to asynchronicity. But one that is equally valid, architecturally speaking, at least based on the results that Oracle seems to be getting from its machines.

Oracle has been selling Exadata parallel database clusters in one form or another for the past three years now, and has 1,000 companies using the boxes with another 3,000 expected by the end of fiscal 2012 next June. Ellison calls this the fastest ramping product in Oracle’s history, which is tough to gauge because Oracle has been around so long it hasn’t had a new class of products since it entered the application market in the early 1990s. The current Exadata X2-2 machines are based eight nodes of two-socket Xeon 5600 servers, which run the Oracle 11g R2 database; if you have fatter databases that are somewhat allergic to clustering, then the Exadata X2-8 has two eight-socket Sun Fire X4800 servers. My understanding is that RAC peters out once you get beyond eight nodes, but good luck finding any documentation on that. Officially, Oracle says you can connect up to eight racks of the Exadata database machines together to create a 64-node monster.

Here’s the first asynchronous processing choice in the Exadata machine. Instead of having each database node have its own storage and be a shared-nothing database or having each node talk out to a shared SAN storage, Oracle created a parallel Exadata storage server that links to the parallel database nodes through a 40 Gb/sec InfiniBand switch. That InfiniBand network is like the system bus of the old AS/400. And those Exadata storage arrays are like IOPs controlling access to storage and helping out with a lot of the I/O that would otherwise be performed by the central processor. Moreover, you don’t have to buy as much raw disk capacity because you can get at least a factor of 10X compression on the data underlying the database.

In this case, a rack of Exadata has up to 14 of these Exadata storage servers, which have four 96 GB flash storage units plugged into the PCI-Express 2.0 buses of the Sun Fire X4170 servers on which they are built. These Exadata storage servers execute SQL queries in parallel and–here’s an important bit–against data that is compressed by a factor of 10 or 15 using an algorithm that Oracle calls hybrid columnar compression. When you combine the data compression with high-speed flash, it takes a lot less compute to do a query and you can move data very quickly across that InfiniBand network to those 11g R2 database nodes at the top of the rack.

This separation of duty was one of the hallmarks of the AS/400, which has largely been reversed by the convergence of the AS/400 with the RS/6000 and IBM’s desire to have central processors do a lot of the I/O work rather than smart IOPs, which incidentally were not supported in AIX. Rather than make AIX asynchronous like on the AS/400, IBM made i5/OS and IBM i centralized, like on the RS/6000. And now, unlike Oracle, it has all of its eggs in the same CPU basket.

On database workloads, Ellison said that customers are seeing anywhere from 10 to 50 times speedup in data warehousing queries and anywhere from four to 10 times for online transaction processing–and it has inherent fault tolerance because it consists of clusters for both storage and database engines.

A rack of fully loaded Exadata X2-2 machines, including Oracle 11g, RAC, and the Exadata storage software, costs $4.5 million at list price; about a quarter of that is hardware costs. It’s about $5.6 million for the Exadata X2-8 with the fat server nodes. Here’s how Ellison stacked the Exadatas up against a top-end Power Systems 795 server with four DS8700 disk arrays feeding it data:

This did not reflect well on the Power 795, as you can see. But you can also see that Ellison was very careful to imply higher transaction and query processing rates without actually getting into details. Moreover, his comparison does not include the cost of software on the machines. That is just for the hardware. With the base Oracle 11g, RAC, and Exadata storage software, those eight racks are more like $35.6 million. It would cost $12.2 million to add the 11g R2 database to the Power 795, plus another $1,500 per core or $384,000 for AIX 7.1. That adds up to $31.4 million at list price, and that is without fault tolerance but with an actual shared memory space for the database, which might be important for OLTP workloads (probably less so for data warehouses).

You have to take it with a grain of salt–and do your math–when Larry says things like this: “The Exadata system costs less than a memory upgrade on the IBM pSeries. And you have to be willing to run a lot faster.” And then he added: “The p795 is one big expensive single point of failure.” Which makes a good point, and which is why I have been arguing for clusters of baby IBM i systems for like 15 years now.

Now, there’s another level of asynchronous processing in the Exa line of machines from Oracle, now that it has fleshed out the product line more last week. Last year at OpenWorld, Oracle debuted the Exalogic parallel middleware clusters, which run a tweaked version of its WebLogic application service and can be used to run Java applications or other app servers as well. You plug Exadata and Exalogic together and you have database and middleware tiers, all tuned up and running on Oracle’s own Linux, which is a variant of Red Hat‘s Enterprise Linux with Oracle’s own variant of the operating system kernel.

Last week at OpenWorld, the Exalytics in-memory business intelligence appliance, which runs parallelized versions of the TimesTen OLTP database and the Essbase multidimensional OLAP database, was also announced, running on a 40-core Sun Fire X4470 and sporting some of the compression used in the Exalogic storage servers. Thanks to that compression, it can hold about 5 to 10 terabytes of compressed database data in its 1 TB of main memory, and can scan a database at about 200 gigabytes per second. With compression on, it scan a 5TB database in about 5 seconds, according to Ellison, who bragged it was “faster than the speed of thought.”

The fourth new machine from Oracle, also announced at OpenWorld this year, is the Big Data Appliance, which runs its Berkeley DB NoSQL database (a form of non-relational database that is used for holding unstructured data like Web logs, Web pages, and such material). This machine, which is based on the Exadata cluster, also runs an Oracle-supported version of Apache Hadoop. This is a program that implements the MapReduce technique of chewing through that unstructured data, a technique first created by Google for its search engine and cloned by Yahoo.

Here’s the thing: Even though three out of the four machines use different databases (one doesn’t have any at all), they are designed to plug together over that InfiniBand network and do different kinds of processing work cooperatively. That’s another layer of asynchronous, specific processing that Oracle is embracing.

But Oracle is not about to stop there. The company launched the first 100 modules of its Fusion line of application software, which takes all the best ideas from its JD Edwards, E-Business Suite, PeopleSoft, Siebel, and other application suites and encapsulates them in Java running atop Oracle’s WebLogic middleware and smacking against its 11g R2 database. The Fusion applications are missing manufacturing and distribution modules, but have supply chain, procurement, human resources, project management, compliance, and financial modules right now. And here’s the kicker: You can run the Fusion apps on general purpose Java machines (like X86 servers or Power Systems), you can run them on Oracle’s engineered systems as outlined above, or you can run them on the Oracle Public Cloud, which was also announced last week.

The Fusion applications were supposed to take four years to develop, and it took six years in part because Oracle needed to put a more collaborative, social network-style interface at the heart of the user interface. You don’t scroll through app screens to use Fusion so much as plough through tasks and have dashboards of information that is relevant for whatever your responsibilities are to the company. Fusion works on PCs, mobile phone, and tables, and it is also available on Oracle’s Public Cloud. And if you have developed your own applications running against the Oracle stack, you can move them to Oracle’s cloud–and you can also move them back. It is all the same software, on premise or on cloud.

It won’t be long before Oracle starts using this slogan: Hardware. Software. Applications. Cloud. Complete.

It’s 1988 all over again, except this time it looks like it’s the Exa System/400.

IBM should be grateful that Oracle has not figured out how to build baby clusters out of microservers that are suitable for SMBs, running these Exa software stacks and Fusion apps. I think eventually Oracle will, in fact. The company would be foolish not to scale the Exa line down to real SMBs, who only want the same kind of system, but with a $25,000 entry hardware price tag, not $330,000 for a quarter rack of hardware with two database nodes and three Exadata storage servers. That’s overkill for an SMB–unless it starts on the cloud and never does anything but use the cloud, which will put the Fusion apps into a virtual machine container for security.

If I were Larry Ellison, I would start pitching the cloud against on premise entry servers or build some baby clusters–probably both. Either could mean big trouble for Big Blue and its Power Systems business.