Guru: Using Curl To Interact With Web Services
January 22, 2018 Jacob Holt
There have been times where I have had to collect or enter data into vendor websites. While usually this was not a hassle, it was an uncommon function of my job. However, it wasn’t long before the standard business process got unbearable. I began looking for ways to automate those tasks by using the command-line program Curl to get and post data to vendor web services.
Curl is an open-source program that supports a plethora of internet protocols for transferring data. With its wide-ranging parameters, this program offers much power for automating tedious tasks. But just like other highly flexible programs, Curl can sometimes be a beast to master. Today we will go through the basics of Curl and how it might be of benefit to you.
To begin working with Curl, you need a web service to test with and a client system with Curl installed. For this example, I use the web service offered by www.jsontest.com, because it does not require me to create an account to use it. If you have access to a Unix or Linux system, you may already have Curl installed. On IBM i systems, IBM already packages Curl and you can run it after calling qp2term from the green screen or ssh’ing into a shell.
curl --url date.jsontest.com
{
"time": "07:50:46 PM",
"milliseconds_since_epoch": 1514749846201,
"date": "12-31-2017"
}
When you run — “curl –url date.jsontest.com” — Curl outputs the data from the site to standard output. There are two ways to redirect this data to a stream file. One way is to use a redirection operator:
curl --url date.jsontest.com > date.txt”
The other is to add the “–output” parameter:
curl --url date.jsontest.com --output date.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100 100 100 0 0 100 0 0:00:01 --:--:-- 0:00:01 243
Running the command like this gives you live information on the progress of the download while outputting the data to date.txt in your working directory.
What About Web Services That Require Tokens For Authentication?
So far we have seen how to send requests to simple services that do not require any form of authentication, but what about the web services that do? NOAA has many web services that do not require tokens, but NOAA’s Climate Data Online web service does. I was able to get a token here and will use the Climate Data Online service as an example. Usually, you will need to pass the token as part of the packet’s header with the parameter “–header” as seen here:
curl --url “https://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:75235&units=standard&startdate=2018-01-01&enddate=2018-01-02” --header token:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --output dallas_airport_station_daily.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 641 0 641 0 0 641 0 0:00:01 --:--:-- --:--:-- 945
The JSON file looks like this (after I got it into pretty print):
{
"metadata": {
"resultset": {
"offset": 1,
"count": 5,
"limit": 25
}
},
"results": [
{
"date": "2018-01-01T00:00:00",
"datatype": "PRCP",
"station": "GHCND:USW00013960",
"attributes": ",,H,2400",
"value": 0
},
{
"date": "2018-01-01T00:00:00",
"datatype": "SNOW",
"station": "GHCND:USW00013960",
"attributes": ",,H,",
"value": 0
},
{
"date": "2018-01-01T00:00:00",
"datatype": "SNWD",
"station": "GHCND:USW00013960",
"attributes": ",,H,",
"value": 0
},
{
"date": "2018-01-01T00:00:00",
"datatype": "TMAX",
"station": "GHCND:USW00013960",
"attributes": ",,H,2400",
"value": 30
},
{
"date": "2018-01-01T00:00:00",
"datatype": "TMIN",
"station": "GHCND:USW00013960",
"attributes": ",,H,2400",
"value": 17
}
]
}
What Are The Pitfalls To Look Out For?
When dealing with https connections, Curl may complain about not being able to validate the TLS certificate and will refuse to complete the connection. This can happen if the client does not have the correct authoritative certificates installed or if the certificate of the web service is self signed. If you trust the certificate, you can override the curl error with the parameter “–insecure”. This doesn’t truly mean that your connection is insecure. Your connection is still encrypted; it just means that Curl cannot verify the legitimacy of the server. Use this “–insecure” parameter sparingly, if at all.
When passing data to Curl parameters, you may have to place the data within single or double quotation marks, depending on how your environment interprets the special characters of some strings. If I were to run the following command in the bash shell, I would have to place both the URL and token (key) — but not the username — in quotations marks:
curl --url "https://example.com/api/people?hair=bald&acquaintance=joeshmoe" \ --header user:joeshmoe \ --header 'key:M@ny$pec!al(haractersth@tb@shw!llnotl!ke'
I have found many uses for Curl, and perhaps you will find it to be of use as well. While we looked at getting data from a web service, it can be used to upload files to FTP sites or even send and receive emails. With its flexibility you can easily get lost. That is where Everything Curl comes in.
Very interesting article… However, I tried your example and it appears that Curl is not available on our v7r1 system. Do I need to download/install it?
IBM supplies Curl with PASE. I have run it on a 7.1 system. Try running a curl command within Qshell.