Guru: Web Services, DATA-INTO and DATA-GEN, Part 1

April 5, 2021 Jon Paris

Many of the “Can you help me with. . . ” communications that cross my desk these days include reference to JSON. Sometimes the questioner is receiving JSON in a file, or has to retrieve it from a web service, or needs to generate JSON in response to a query. While there are many ways to handle these requirements, RPG’s built-in DATA-INTO and DATA-GEN can do a lot of the heavy lifting for you and are quite simple once you understand the basics.

In this series of tips, I am going to start with a basic example that uses both DATA-INTO and DATA-GEN to generate the request data and processes the response from a simple REST web service. Subsequent tips in the series will delve further into the capabilities and quirks of these opcodes by working through more complex examples. In addition to the RPG opcodes, I will be using Scott Klement’s YAJL library – specifically the YAJLDTAGEN and YAJLINTO routines. I will also be using Scott’s HTTPAPI library for communicating with the web service. While I will give some details of using HTTPAPI, that is not my primary purpose in these tips, so if you are interested in knowing more about this fabulous tool let me know.

Time to introduce the web service. In these early examples I will be using the free “fake” REST services provided by the website JSONPlaceholder. I chose this site because it is likely to still be there in a year or two when those who read this in the future may want to try the code. I also like the fact that you do not need an account or any authorization code to use it. As a result, you should be able to copy my code and make your own explorations at any time.

This story contains code, which you can download here.

This initial example uses a web service that will create a new blog post (consisting of a title and the blog entry content) for a specified author (the userid). The service will return a confirmation which includes a unique ID assigned by the service to the article. This service uses the POST method and the data must be sent in JSON format in the HTTP request body. The actual request needs to be in the format shown below.

{
    "title":"Post title goes here ...",
    "body":"This is the post content ...",
    "userId": 1
}

Below is a sample of the JSON response that the web service returns. Note that it basically contains just the data that I submitted together with the unique ID that the service assigned to the new post. I should point out at this stage that, because this is a “fake” service, the ID it returns is always the same and the data is not actually being stored. So don’t be alarmed if you test this and it doesn’t seem to be working as you might expect in the real world!

{
  "title": "Post title goes here ...",
  "body": "This is the post content ...",
  "userID": 1,
  "id": 101
}

Obviously, in a simple case like this I could form the JSON request by simply stringing the content together. But that’s messy and error prone so let’s look at how DATA-GEN can do the work for us. Besides, it’s rare that you’ll have a request this simple in real life.

The first thing to do is to define a data structure (DS) that maps to the desired JSON data. In this case it is just a simple single level DS containing the three required fields in their correct sequence. That DS will then be processed by DATA-GEN, with the help of Scott’s YAJLDTAGEN routine, to produce the required JSON. Here’s the DS definition.

<A>   Dcl-DS  requestData  Qualified  Inz;
<B>      title    varchar(30);   // Blog post title
         body     varchar(200);  // Blog post content
         userid   int(5);        // User ID of the post author
      End-Ds; 

Note that the sequence of fields in the DS and their names must match the sequence and exact spelling of the JSON elements to be created. RPG would treat the field names Title, TITLE and title as being the same, JSON does not! Names in JSON are case-sensitive! So you must use the exact spelling, including case.

<A> The DS is specified as QUALIFIED so that I can use the same field names in the response DS, as you will see in a moment.

<B> The fields in the DS have the names required by the web service. title and body are both defined as Varchar to ensure that we don’t generate a large number of trailing spaces in the fields. The userid (think customer number) is specified as int but could be any numeric data type.

Once we have the DS in place we can tell DATA-GEN to generate JSON from it. Here’s how that looks:

   // Generate JSON payload
<C> Data-Gen requestData
<D>          %Data( request )
<E>          %Gen( 'YAJL/YAJLDTAGEN' ); 

<C> Identifies our requestData DS as the source of the data.

<D> Names the target field to receive the generated JSON. In my example it is a varchar field with a length of 200 characters.

<E> %GEN identifies that the YAJLDTAGEN generator is to be used. It also allows for the specification of additional optional parameters that are passed to the generator, but none are needed in this example. You will see them in action in future tips.

That’s all we need in this simple example. If you study the complete code (which you can download here) you will see that I used the DSPLY opcode to obtain the data that is to be passed to the web service.

Once the JSON request has been built it is just a matter of sending it to the web service with HTTPAPI. I am using the HTTP_string routine to do this as it is the easiest one to use and the only data I am interested in is the response ID. If the web service call fails, it is sufficient to know there was an error. I am not concerned with the cause. In fact this particular service probably wouldn’t tell me anything useful anyway.

   // Perform POST operation to add the new item
<F> Monitor;
<G>   response = HTTP_string( 'POST' :
<H>                            url:
<I>                           request:
<J>                           'application/json');

Because HTTP_string causes an exception to be thrown, as opposed to returning an error response code, it is normal to run it in a MONITOR group <F>. Any errors detected will cause an exit to the ON-ERROR code which you can see if you study the full example code.

At <G> we identify the variable response as the recipient of the data and that this web service uses the POST HTTP method.

<H> Next we identify the variable that contains the URL for the service we are using.

<I> The next parameter specifies that request is the variable containing the HTTP request body. In this case it is the JSON data that DATA-GEN constructed for us back at <C>.

<J> The final parameter identifies that the request body contains JSON. If the web service additionally supported XML as an input type, we could construct the request in XML format and code this as ‘application/xml’. It is the web service that determines the formats of data that are acceptable. In this case JSON is the only option.

Once HTTP_string has successfully run, we have the JSON data from the web service in the response variable. We can now process it using DATA-INTO. As was the case with DATA-GEN we will need to define a DS. This time though it will receive the values parsed out of the JSON response stream. Here is the DS I am using:

    // DS to receive response data extracted by DATA-INTO
    Dcl-DS  responseData  Qualified  Inz;
<K>    id       int(5);          // Identifying ID for the post 
       title    varchar(30);     // These three should contain same
       body     varchar(200);    //   values as we submitted
       userid   int(5);          //   on the original POST request
End-Ds;

As you can see this DS is basically the same as the one used earlier to generate the JSON request. The exception is the addition of the id field, which the web service generated to uniquely identify the new post <K>. With the DS in place and the web service response in hand we can now use DATA-INTO to parse the JSON into the DS for subsequent processing by our RPG code.

Note that I have deliberately defined the id field in a different position to where it appears in the JSON. This demonstrates that, just like with XML-INTO, DATA-INTO does not require the field sequence to match. It is storing values based on the field names, not on their sequence.

<L> Identifies the responseData DS as the target for the DATA-INTO operation.

<M> %DATA identifies the source of the data to be processed. In this case that is the field response that was the result of the HTTP_string operation at <G>. The ‘case=any’ option specifies that the field names in the JSON should be converted to upper case before attempting to match them against names of the fields in the target DS. Those of you familiar with XML-INTO will recognize this option as being identical in effect to the same option on the %XML BIF. As you will see in future examples, this is true of many of the options used with DATA-INTO.

<N> Last, but not least, we use %PARSER to specify that we will use the YAJLINTO parser to handle the JSON. Just as with %GEN, %PARSER also allows for additional parameters which we will see in action later in the series.

   // All OK if we get this far so extract data and report
<L> Data-Into responseData
<M>           %Data( response : 'case=any' )
<N>           %PARSER( 'YAJL/YAJLINTO' );  // Extracted data now in responseData

And that is all there is to it.

Before I Go

I thought I would take a couple of minutes to address two questions that may have already popped into your mind.

1) “Can’t I use SQL to do all of this kind of JSON parsing for me?”

2) “Doesn’t IBM offer a web services client as part of the Integrated Web Services software?”

The answer to the first question is “Yes, but I’ll leave that explanation to someone else.” I will add though that while I find SQL easy enough to use for simple examples like this, I find that it often becomes horribly complicated and hard to maintain for more complex examples. I am not a fan of the “SQL is the answer — now what was the question?” mentality that seems to have flourished in recent years. The RPG tooling in this area is good and fast.

The answer to the second question is “Yes, but I don’t like it.” It would take far too long for me to explain in detail, but to me using it just doesn’t “feel” like RPG. HTTPAPI was written by an RPG user and it shows. IBM’s offering is a port of a Unix-style Apache tool and involves creating and compiling C stubs to interface to the system components. HTTPAPI also has a huge user following and an active forum where users constantly trade tips and techniques. The same is not true of the IBM offering. Just my perspective — your mileage may vary.

Next Time

In the next tip in this series, I will be taking a look at an example that uses the GET method to retrieve information about blog posts.

In the meantime, if you have any questions or if there are any particular aspects of this very broad topic you would like me to delve into in future tips please let me know.

Jon Paris is one of the world’s foremost experts on programming on the IBM i platform. A frequent author, forum contributor, and speaker at User Groups and technical conferences around the world, he is also an IBM Champion and a partner at Partner400 and System i Developer. Until Covid-19 messed everything up he hosted the RPG & DB2 Summit with partners Susan Gantner and Paul Tuohy. These days he has to be content with everything being done over Zoom. That includes the free Summit Lunch & Learn series, the Summit Hands-On Live! Workshops, and the Virtual RPG & DB2 Summit.

RELATED STORY

RPG’s New DATA-INTO