Infoview Systems Launches Lightweight CDC for IBM i
February 13, 2023 Alex Woodie
IBM i shops that have a need to extract the latest transactions hitting their Db2 for i database for downstream CRM, analytics, or fraud detection use cases may want to check out the new change data capture (CDC) offering unveiled recently by Infoview Systems. When coupled with a streaming data platform like Apache Kafka, CDC opens up new possibilities.
Infoview Systems is a longtime IBM i product and services firm based in Michigan with a global customer base. The company has been active in the data and application integration space for years, first with the sale and implementation of message queuing systems like WebSphere MQ and webMethods for IBM i customers, and more recently with the new generation of event-based messaging middleware, such as Apache Kafka and MuleSoft.
A company recently came to InfoView with a need to pull transactions off its IBM i ERP system to send to an e-commerce system for processing. The customer wanted to minimize overhead on the IBM i system by extracting the transactions from the database as soon as they were written, and moving them into a cloud-based database serving the e-commerce application.
Infoview assessed the various CDC offerings from major vendors on behalf of the customer, says Dmitriy Kuznetsov, vice president of integration and cloud solutions at Infoview. Unfortunately, none of the CDC offerings on the market could support the customer’s requirement.
A big technical hurdle to overcome was the fact that the customer was running a very old system with a denormalized table structure that contained upwards of 1,000 records per file, Kuznetsov says.
InfoCDC simplifies streaming data from IBM i to streaming systems like Kafka. (Image courtesy Infoview Systems)
“It had everything – addresses, customer balances, all kinds of information about customers in the single table,” he tells IT Jungle. “What they needed is to expose is just a small subset of that information.”
Unfortunately, the existing CDC solutions were ill-equipped to handle the customer’s needs. They took an “all or nothing approach,” Kuznetsov says, requiring the customer to replicate the entire record, adding additional network bandwidth and extra heavy lifting.
“There are a few other CDC solutions that are available,” he says. “But they felt that the solutions were either too complex or too expensive or too inflexible when it comes to configuring them just to include a subset of columns, a subset of tables, and having better control over how and where those changes would flow.”
That was the primary motivation behind Infoview’s decision to build its own CDC product, he says. The company developed the offering, tested it on the customer, and now it’s ready to began marketing it more widely.
InfoCDC, as the product is called, is developed in native RPG, along with SQL. It works by monitoring IBM i journals, detecting changes that match the specified criteria, and then streaming it through IBM i data queues to Kafka, MuleSoft, or any other suitable integration platform.
“It is our own IP,” Kuznetsov says. “We built it from ground up, all the logic. But of course, it uses standard IBM i technology. We’ll listen for journal entries using the receive journal entry command, and we just created our exit programs for that and the whole management piece around it.”
The main selling point to InfoCDC is that the product takes a more lightweight approach to CDC than other products on the market from vendors like IBM and Informatica, Kuznetsov says.
“Other products are very comprehensive, and they indeed support use cases where you have hundreds of tables, multiple libraries,” he says. “You can automatically discover and replicate all these tables, all this data to external databases.”
In addition to giving users more options to filter which records they want to capture and push through the data queue to the integration middleware, InfoCDC is also easier to use.
“You don’t have to be PhD in that product,” Kuznetsov says. “It’s basically a plug-in solution. And it’s relatively inexpensive.”
The InfoCDC offering sports a basic greenscreen interface. The company plans to continue developing the solution by adding more components and connectors that make it more of an “end to end” solution, including possibly a REST API.
As a services company, Infoview Systems is also more than happy to help integrate InfoCDC for IBM i shops. The company, which has about 50 workers, employs experts who can help set up the middleware environment, whether it’s open source Apache Kafka, Confluent Cloud, MuleSoft, or Kafka running on AWS or Microsoft Azure.
Once the InfoCDC product is configured and it’s pulling the latest real-time data out of Db2 for i and loading into a Kafka topic, it can be piped just about anywhere. There are dozens, if not hundreds, of pre-built connectors for source and target systems available via Kafka Connect. When the business use case requires the freshest data be available, this is the fastest way to get there, Kuznetsov says.
“Streaming is the method for delivering all these numerous events from IBM i,” he says. “The use case is typically present themselves in a way that is almost like there is no alternative. So if you need some sort of messaging infrastructure and you find that traditional, legacy MQ-based solutions are not scaling well or getting too expensive, the operational overhead is larger, and maybe they’re not as nimble when it comes to adding new channels, supporting new event stream – that’s where companies realized that Kafka is a leader in that space.”
For more information on InfoCDC and its other products, check out the company’s website at www.infoviewsystems.com.
RELATED STORIES
Kafka, JSON, DevOps: Future Proof Your IBM i With Secure, High Performance APIs
Firm Brokers MuleSoft’s Passage Into IBM i World
