Friday, 1 August 2014

How to Invoke REST Web services from DataStage Designer


 The blog provides the information on new capability of DataStage Designer and few  
  example usages

 InfoSphere Information Server Designer provides a stage called Hierarchical Data stage  
 (which was called XML Stage in prior releases to V11.3) that has the capability to parse  
 and compose the hierarchical data formats like JSON and XML. Along with that, it  
  provides the capability to invoke the REST Web services.

 It supports the REST Web services that are configured in different ways as below:
 1. That provides the responses in different formats like JSON, XML, HTML, JPEG, Audio  
 2. Supports different authentication mechanisms like BASIC, DIGEST, LTPA, OAUTH
 3. Configured with SSL (Server and Client authentication)
 4. Headers and Cookies
 5. Different HTTP methods like GET, POST, PUT, DELETE, PATCH etc.

 There is so much data on the social media sites, which you want to pull, transform and 
 send for the  analytics. Here I am taking an example of the IBM Facebook page. From 
 which, I want to retrieve the details for how many likes does IBM page have, which 
 all websites talk about IBM, To fulfill the above requirement, You can use our new 
 capability called REST step from the Hierarchical Datastage stage in DataStage Designer.

  Extracting data from Facebook Page
  The Figure 1 illustrates the DataStage job which retrieves the data from Facebook using 
  REST step  and parses the response data from Facebook to fetch the required data for the

 Figure 1: Extract_And_Parse_DataFromFaceBookPage


The figure 2 shows the assembly design of the Hierarchical datastage. The data
is obtained from Facebook by invoking the REST API using REST step. As the 
data provided by Facebook is in format of JSON, JSON parser step is used to parse 
the data. The REST Step “GetFBPageData” and JSON Parser step called 
“ParseFBData” are added to the Assembly Outline.

Figure 2 : Assembly Editor Design

 The Rest Step “GetFBPageData“ is configured as below:

1.In the General tab: The HTTP Method “GET” is selected , URL of the IBM Facebook
   page is mentioned in the “URL” field. The URL is as 
   in  figure 3.
2. In the Security tab: As Facebook is configured with SSL, select the checkbox Enable the  
    the SSL and Accept the Self signed certificate as shown in figure 4.
3. In the Request tab: Facebook returns the response in different formats. Here we need in  
    the JSON format. So Specify the Content-Type as application/json under the custom 
    header. As shown in figure 5.
4. In the Response tab : Select the check box “ Pass the received body to” and a radio
    button   “A text node named body in the Output Schema” and specify text/javascript  
    under  the content type as shown in figure 6.

Figure 3: General tab of Rest Step "GetFBPageData"
 Figure 4 : Security tab of Rest Step “GetFBpageData”
 Figure 5 : Request tab of Rest Step “GetFBPageData”
 Figure 6 : Response tab of Rest Step “GetFBPageData”
The Output of the REST call which is invoked in the “GetFBPageData” REST step is available in the body element in the output schema tab. The output schema of the REST step is as in figure 7.

Figure 7 : The Output Schema of the Rest Step "GetFBPageData"

 The JSON parser step “ParseFBData” is configured as below:
1.Under the JSON Source tab : Select the String set option, and from the drop down 
   select the bodyelement coming from the REST step “GetFBPageData” as shown in the   
   figure 8.  
2.Under the Document root: Browse and select the schema which conforms with the json
   data retrieved from earlier REST step as shown in figure 9.  
3.Under Validation tab: Minimal Validation is selected by default.  

Figure 8 : JSON Source tab of JSON Parser step “ParseFBData”  

 Figure 9 : Document root of JSON Parser step “ParseFBData”

The Output step is configured as below: 

Under the Mappings tab, the target link “DETAILS” is mapped to the top to fetch the parsed details of IBM Facebook page as shown in figure 10.

Figure 10 : Mappings tab of Output step
Compile and Execute the job to fetch the required details of the IBM Facebook page shown in figure 11:

Figure 11 : Data from IBM facebook page.
The output describes the details like ID of the Page, what it is about , when it is founded, 
 how many likes it got, the count talking about this page, username, websites where it is  
 talked etc. This data can be used in the analytics.

Conclusion : The REST capability in the hierarchical Datastage stage can be used to fetch 
 the data like social media data from different applications like Facebook, LinkedIn, 
 Twitter etc which exposes the services using REST

We have articles published on developer works for the integration scenarios with SOFT 
Layer, Cloudant and Information Governance Catalog Glossary using DataStage. 

 Disclaimer: “The postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.” 


  1. How do we achieve the same REST webservice access in Version 9.1?

    1. Hi Ravi,

      The REST Web Service is the new feature introduced in the IS Version 11.0, hence its not possible to access REST WebServices using IS Version 9.1.

      Thank you

    2. Hi Ravi,

      Were you able to figure out how to implement this in 9.1? I see some posts online which say that 9.1. supports it but not sure exactly how to go about it.

      Please let me know if you have any inputs on this.


  2. Hi Ravi, Could you please send or post the JSON file sample. I was trying to parse the file, but the unable to map the fields. Where will you find the schema for the JSON files which you get from the facebook or twitter.
    Thank you

  3. I have one confusion, regarding fig 9 "Document root of JSON Parser step “ParseFBData”. From where we get the JSON Schema file if I want to do the same.

    1. You need to used the same JSON file under document root. The file can be taken from browser

    2. can you give a few simple steps as to HOW it canbe gotten from the browser?

    3. Did anyone get an answer to this?

  4. Thank u for giving this information. It is saving my valuable time. Data Stage Online Training

  5. Hi Sir.

    The article is great sir - I am trying to replicate the same.
    however regarding fig 9 "Document root of JSON Parser step “ParseFBData”.

    could you please provide the JSON shema file as an attachment.

    I don't know how to create the JSON schema file from the browser.

    Kindly help us

  6. I am unable to View Figure 5 and Figure 7 in the above doc.

    Could you please provide me Doc

  7. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Datastage, kindly contact us
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    Ph:(0) 9738075708 / 080 - 41103383

  8. Thank you for sharing this powerful article, your explanation is clear and very easy to understand. Please kindly visit our site to get more information about IT solution.
    Melbourne SEO Services

  9. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

  10. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

  11. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

  12. I read this article. I think You put a lot of effort to create this article. I appreciate your work.
    thesis Writing Service