Skip to main content
Question

HTTPCaller POST passing attributes within Body not working, any help?

anamanvil
Forum|alt.badge.img

Hi All,

 

It is often when I need to make used of HTTPCaller transformer to call another FME Workspace hosted on FME Server.

 

I commonly pass all attributes using Query String Parameters. Unfortunately, this time this method doesn't seem to work as my URI gets too long due to complex geometry stored as WKT.

 

I am trying to find a way to upload these parameters as part of the Body instead as Content Type application/json but I might have something wrong as even though it runs the workspace is not able to retrieve these attribute values from the transformer.

 

I know how to do this using Python but I would like to avoid this and use HTTPCaller transformer if possible.

 

I would appreciate any help or suggestion.

 

Thanks,

Ana

 

18 replies

david_r
Celebrity
  • david_rCelebrity
  • October 26, 2020

Can you post a minimal workspace here that shows the HTTPCaller and all the associated settings?

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020

Hi @david_r​ , I have just uploaded a HTTPCallerDummy workspace so you can see the settings I currently have.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • October 27, 2020

Out of curiosity why are you using the HTTPCaller instead of the FMEServerJobSubmitter?

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
jdh wrote:

Out of curiosity why are you using the HTTPCaller instead of the FMEServerJobSubmitter?

Hi @jdh​, I am doing so because I need to retrieve the output of the workspace I run which I believe is not possible by using FMEServerJobSubmitter. Correct me if I am wrong. Using the HTTPCaller I am able to retrieve the output as JSON and continue the data transformation process.

david_r
Celebrity
  • david_rCelebrity
  • October 27, 2020
anamanvil wrote:

Hi @jdh​, I am doing so because I need to retrieve the output of the workspace I run which I believe is not possible by using FMEServerJobSubmitter. Correct me if I am wrong. Using the HTTPCaller I am able to retrieve the output as JSON and continue the data transformation process.

You may want to either use, or let yourself be inspired by, the FMEServerDataDownload custom transformer available on the FME Hub:

https://hub.safe.com/publishers/giosp/transformers/fmeserverdatadownload

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
anamanvil wrote:

Hi @jdh​, I am doing so because I need to retrieve the output of the workspace I run which I believe is not possible by using FMEServerJobSubmitter. Correct me if I am wrong. Using the HTTPCaller I am able to retrieve the output as JSON and continue the data transformation process.

thanks @david_r​, I am calling this as Data Streaming Service though. So not sure if this would still help me but I will take a look definitely.

 

Is there any way though I could use HTTPCaller but passing the attributes as part of the body to make the POST request?

 

As mentioned before, I have uploaded a workspace dummy with the parameters I am using to make the call. So maybe this gives you the hint as to what I am doing wrongly.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • October 27, 2020
anamanvil wrote:

Hi @jdh​, I am doing so because I need to retrieve the output of the workspace I run which I believe is not possible by using FMEServerJobSubmitter. Correct me if I am wrong. Using the HTTPCaller I am able to retrieve the output as JSON and continue the data transformation process.

Not directly no, the FMEServerJobSubmitter only returns the job ID. The FMEServerLogFileRetriever will return the log file which can be parsed to get the output location, but at that point you probably are better using the rest api directly instead.

 

david_r
Celebrity
  • david_rCelebrity
  • October 27, 2020

I believe that when you invoke the fmedatastreaming service, the message body of your POST request is sent as input to the workspace reader, and not automatically parsed as published parameters (as is the case for the "regular" REST API).

See: https://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/service_datastreaming.htm

This means that you'll have to use a Textfile reader in your workspace to pick up the message body contents, then something like the JsonFlattener to get the srid and wkt attributes in the JSON you're posting. Alternatively you could also use a JSON reader (but personally I find it easier to debug with a Textfile reader).

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
david_r wrote:

I believe that when you invoke the fmedatastreaming service, the message body of your POST request is sent as input to the workspace reader, and not automatically parsed as published parameters (as is the case for the "regular" REST API).

See: https://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/service_datastreaming.htm

This means that you'll have to use a Textfile reader in your workspace to pick up the message body contents, then something like the JsonFlattener to get the srid and wkt attributes in the JSON you're posting. Alternatively you could also use a JSON reader (but personally I find it easier to debug with a Textfile reader).

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

david_r
Celebrity
  • david_rCelebrity
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

How did you configure the request in Postman compared to the HTTPCaller? Could you post a screenshot of the Postman setup?

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

   Same as this one @david_r​ 

 Capture2Capture1

david_r
Celebrity
  • david_rCelebrity
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

Thanks, I see the difference. For the same setup as in Postman, you'll have to configure the HTTPCaller with upload data set to "Multipart / Form Data", then specify the key/value pairs in the "Multipart Upload" pane below:

2020-10-27 16_02_36-HTTPCaller Parameters

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

Thanks @david_r but I tried that before making this enquiery and it didn't work. So that is way I approached the idea of sending it as Json within the Upload Body.

david_r
Celebrity
  • david_rCelebrity
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

What error message do you get?

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 27, 2020
anamanvil wrote:

Thanks @david_r​  this might be the reason. However, how is it that Postman is able to replicate the same call I make with the HTTPCaller and retrieve the workspace output whereas FME cannot? Postman is calling Data Streaming Service as well so I believe there might be a way to replicate that on FME without the need of modifying the workspace that get called something which I would like to avoid as these data streaming workspaces are being called from various side, not only FME.

@david_r​, not an error at all. I am able to run the workspace but still having the issue where no attributes is being passed through the call. So I have my custom JSON response telling me it has not valid WKT or Order Id and therefore the workspace is terminated.

 

The below is how I have tried. Also using using application/x-www-form-urlencoded which is the content type my Postman call is using.

 

Capture3

david_r
Celebrity
  • david_rCelebrity
  • October 28, 2020

First: Upload Data in the HTTPCaller should be set to "Specify upload body", not "Multipart / form data", sorry for the confusion.

 

Secondly: The most important thing to know is that there is no automatic mapping between the upload body and any published parameters in the workspace when using the fmedatastreaming service. This is really important, because the only way to retrieve the contents from the upload body is through a text (or similar) reader in the workspace that is being called. This is also mentioned in the documentation that I linked somewhere else in this thread.

 

anamanvil
Forum|alt.badge.img
  • anamanvil
  • Author
  • October 28, 2020
david_r wrote:

First: Upload Data in the HTTPCaller should be set to "Specify upload body", not "Multipart / form data", sorry for the confusion.

 

Secondly: The most important thing to know is that there is no automatic mapping between the upload body and any published parameters in the workspace when using the fmedatastreaming service. This is really important, because the only way to retrieve the contents from the upload body is through a text (or similar) reader in the workspace that is being called. This is also mentioned in the documentation that I linked somewhere else in this thread.

 

​Hi @david_r​, you are absolutely right. Somehow I thought I could replicate what Postman was doing but I was wrong.

 

Yesterday, I started investigating further the information you sent and I was able to solve the issue by just adding a Text File Reader within the FME Workspace I was planning to make the POST Call.

 

So many thanks for your help and time :)

david_r
Celebrity
  • david_rCelebrity
  • October 28, 2020
anamanvil wrote:

​Hi @david_r​, you are absolutely right. Somehow I thought I could replicate what Postman was doing but I was wrong.

 

Yesterday, I started investigating further the information you sent and I was able to solve the issue by just adding a Text File Reader within the FME Workspace I was planning to make the POST Call.

 

So many thanks for your help and time :)

Am glad to hear! Regarding Postman, one possibility could be that there were default values defined in the published parameters.

Reply


Helpful Members This Week

Community Stats

31,158
Posts
118,135
Replies
38,784
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.