Friday 1 August 2014

How to Invoke REST Web services from DataStage Designer

 Overview:

 The blog provides the information on new capability of DataStage Designer and few  
  example usages

 InfoSphere Information Server Designer provides a stage called Hierarchical Data stage  
 (which was called XML Stage in prior releases to V11.3) that has the capability to parse  
 and compose the hierarchical data formats like JSON and XML. Along with that, it  
  provides the capability to invoke the REST Web services.

 It supports the REST Web services that are configured in different ways as below:
 1. That provides the responses in different formats like JSON, XML, HTML, JPEG, Audio  
     etc.
 2. Supports different authentication mechanisms like BASIC, DIGEST, LTPA, OAUTH
 3. Configured with SSL (Server and Client authentication)
 4. Headers and Cookies
 5. Different HTTP methods like GET, POST, PUT, DELETE, PATCH etc.

 Scenario:
 There is so much data on the social media sites, which you want to pull, transform and 
 send for the  analytics. Here I am taking an example of the IBM Facebook page. From 
 which, I want to retrieve the details for how many likes does IBM page have, which 
 all websites talk about IBM, To fulfill the above requirement, You can use our new 
 capability called REST step from the Hierarchical Datastage stage in DataStage Designer.

  Extracting data from Facebook Page
  The Figure 1 illustrates the DataStage job which retrieves the data from Facebook using 
  REST step  and parses the response data from Facebook to fetch the required data for the
 analytics.


 Figure 1: Extract_And_Parse_DataFromFaceBookPage

 












The figure 2 shows the assembly design of the Hierarchical datastage. The data
is obtained from Facebook by invoking the REST API using REST step. As the 
data provided by Facebook is in format of JSON, JSON parser step is used to parse 
the data. The REST Step “GetFBPageData” and JSON Parser step called 
“ParseFBData” are added to the Assembly Outline.

Figure 2 : Assembly Editor Design

 The Rest Step “GetFBPageData“ is configured as below:

1.In the General tab: The HTTP Method “GET” is selected , URL of the IBM Facebook
   page is mentioned in the “URL” field. The URL is https://graph.facebook.com/IBM as 
   in  figure 3.
2. In the Security tab: As Facebook is configured with SSL, select the checkbox Enable the  
    the SSL and Accept the Self signed certificate as shown in figure 4.
3. In the Request tab: Facebook returns the response in different formats. Here we need in  
    the JSON format. So Specify the Content-Type as application/json under the custom 
    header. As shown in figure 5.
4. In the Response tab : Select the check box “ Pass the received body to” and a radio
    button   “A text node named body in the Output Schema” and specify text/javascript  
    under  the content type as shown in figure 6.

Figure 3: General tab of Rest Step "GetFBPageData"
 Figure 4 : Security tab of Rest Step “GetFBpageData”
 
 Figure 5 : Request tab of Rest Step “GetFBPageData”
 Figure 6 : Response tab of Rest Step “GetFBPageData”
The Output of the REST call which is invoked in the “GetFBPageData” REST step is available in the body element in the output schema tab. The output schema of the REST step is as in figure 7.

Figure 7 : The Output Schema of the Rest Step "GetFBPageData"


 The JSON parser step “ParseFBData” is configured as below:
1.Under the JSON Source tab : Select the String set option, and from the drop down 
   select the bodyelement coming from the REST step “GetFBPageData” as shown in the   
   figure 8.  
2.Under the Document root: Browse and select the schema which conforms with the json
   data retrieved from earlier REST step as shown in figure 9.  
3.Under Validation tab: Minimal Validation is selected by default.  

Figure 8 : JSON Source tab of JSON Parser step “ParseFBData”  
 

 Figure 9 : Document root of JSON Parser step “ParseFBData”


The Output step is configured as below: 

Under the Mappings tab, the target link “DETAILS” is mapped to the top to fetch the parsed details of IBM Facebook page as shown in figure 10.

Figure 10 : Mappings tab of Output step
 
Compile and Execute the job to fetch the required details of the IBM Facebook page shown in figure 11:

Figure 11 : Data from IBM facebook page.
 
 
The output describes the details like ID of the Page, what it is about , when it is founded, 
 how many likes it got, the count talking about this page, username, websites where it is  
 talked etc. This data can be used in the analytics.

Conclusion : The REST capability in the hierarchical Datastage stage can be used to fetch 
 the data like social media data from different applications like Facebook, LinkedIn, 
 Twitter etc which exposes the services using REST

We have articles published on developer works for the integration scenarios with SOFT 
Layer, Cloudant and Information Governance Catalog Glossary using DataStage.

 http://www.ibm.com/developerworks/data/library/techarticle/dm-1407governrest/index.html 

 Disclaimer: “The postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.” 




37 comments:

  1. How do we achieve the same REST webservice access in Version 9.1?

    ReplyDelete
    Replies
    1. Hi Ravi,

      The REST Web Service is the new feature introduced in the IS Version 11.0, hence its not possible to access REST WebServices using IS Version 9.1.

      Thank you

      Delete
    2. Hi Ravi,

      Were you able to figure out how to implement this in 9.1? I see some posts online which say that 9.1. supports it but not sure exactly how to go about it.

      Please let me know if you have any inputs on this.

      Thanks

      Delete
    3. Hi Ravi, Could you please send or post the JSON file sample. I was trying to parse the file, but the unable to map the fields.
      send to mail

      Delete
  2. Hi Ravi, Could you please send or post the JSON file sample. I was trying to parse the file, but the unable to map the fields. Where will you find the schema for the JSON files which you get from the facebook or twitter.
    Thank you

    ReplyDelete
  3. I have one confusion, regarding fig 9 "Document root of JSON Parser step “ParseFBData”. From where we get the JSON Schema file if I want to do the same.

    ReplyDelete
    Replies
    1. You need to used the same JSON file under document root. The file can be taken from browser

      Delete
    2. can you give a few simple steps as to HOW it canbe gotten from the browser?

      Delete
    3. Did anyone get an answer to this?

      Delete
    4. call the URL from browser and then take the returned json and put it into file .json and then you can brows from DS designer from Document root tab as in fig 9 to your file system and choose the .json file that you have just created.

      Delete
  4. Thank u for giving this information. It is saving my valuable time. Data Stage Online Training

    ReplyDelete
  5. Hi Sir.

    The article is great sir - I am trying to replicate the same.
    however regarding fig 9 "Document root of JSON Parser step “ParseFBData”.

    could you please provide the JSON shema file as an attachment.

    I don't know how to create the JSON schema file from the browser.

    Kindly help us

    ReplyDelete
    Replies
    1. call the URL from browser and then take the returned json and put it into file .json and then you can brows from DS designer from Document root tab as in fig 9 to your file system and choose the .json file that you have just created.

      Delete
    2. which URL and how to take returned json?

      Delete
  6. I am unable to View Figure 5 and Figure 7 in the above doc.

    Could you please provide me Doc

    ReplyDelete
  7. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Datastage, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    MaxMunus
    E-mail: sangita@maxmunus.com
    Ph:(0) 9738075708 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete
  8. Thank you for sharing this powerful article, your explanation is clear and very easy to understand. Please kindly visit our site to get more information about IT solution.
    Melbourne SEO Services

    ReplyDelete
  9. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

    ReplyDelete
  10. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

    ReplyDelete
  11. the blog is good and Interactive it is about Mulesoft API Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online training

    ReplyDelete
  12. How to extract data from Google BigQuery useing Datastage?
    Does anyone here successfully accomplish this?

    ReplyDelete
  13. This Blog Provides Very Useful and Important Information. I just Want to share this blog with my friends and family members. Mulesoft certification training

    ReplyDelete
  14. Hi Shweta,

    It is a very nice blog.. Thanks for sharing the info.
    Need one more help - "How to Invoke LinkedIn data from DataStage Designer" while creating linkedin app I was unable to get all the info which I have to fill. could you please guide me..
    Find below my contact details,

    Thanks
    Sudipta
    9903895374
    ssudipta4@gmail.com


    ReplyDelete
  15. I'm getting this error:

    "text/javascript does not match the received content type: application/json"

    In the step: Figure 6 : Response tab of Rest Step “GetFBPageData”
    json option doesn't there

    ReplyDelete
  16. Hi,
    I exactly followed these steps which was a great help indeed. I ran the job and it executed successfully but I do not get any output. Will any of you please help me. This is really important and urgent for me.

    Thanks,
    Arpita (arpitasatarkar@gmail.com)

    ReplyDelete
    Replies
    1. Did u get any help on this I am also facing the same issue.

      Delete
    2. Did u get any help on this I am also facing the same issue.

      Delete
    3. i am also getting same error. Are you able to run successfully?

      Delete
    4. en mi caso fue que estaba colocando la url sin la s en el http, supongo q es un tema de seguridad en el servidor y no regresa nada si no le pones https

      Delete
  17. I think there is a high need of providing some extra information about Power BI since it is one of the most easy tool being used nowadays for solving complex IT problems.

    powerbi read rest

    ReplyDelete
  18. i am getting connection reset java error while calling the rest api how can i solve this issue?

    ReplyDelete
  19. Your blog is in a convincing manner, thanks for sharing such an information with lots of your effort and time datastage online training

    ReplyDelete
  20. Thank you for sharing that how can we invoke rest web services from database. This is very helpful. Thank you for sharing this with us.

    API based integration

    ReplyDelete
  21. It was very helpfull for beginners like me.Thaks much for sharing,COuld you please share thesimilar one for POST method to verify the authentication with username and password.I have tried to it in bodyfilepath ,traffic was going to target but it has no request body ,so empty login .Can some one try to help to configure it in 11.7 version

    ReplyDelete
  22. Can someone please tell steps to connect to web services using OAuth2bearer token ? On each run there should be new token

    ReplyDelete
  23. Very Good Post! Thank you so much for sharing this good post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Visit us: Java Online Training Hyderabad
    Visit us: Java Training

    ReplyDelete