Solved

How can I successfully POST json data to FME-server using REST?

  • 20 July 2018
  • 9 replies
  • 88 views
K
Badge +3

Goal

I want to use FME-server as endpoint for sensordata (senML). This sensordata is posted from the sensors to our server with JSON as body. 

Situation

I created a workspace which is comparable with the posts as described in these posts and everyting related to this

The last URL also contains my question from earlier today, but I got advised to create a new topic.

In short my workspace contains:

  •  a textfile reader with the setting 'read the whole file at once'
  • some transformers to interprete the json
  • postgis writer

The workspace works perfectly when running from my computer with FME workbench.

I uploaded the workspace to FME-server, using the jobsubmitter as service and in the settings of the service I pointed to the reader (Send HTTP Message Body to Reader)

Problem/testing

I'm using Postman to test posting the data. My http post contains the required headers and token certification, so this is no issue. In my FME-server job I can see that it is trying to run the job. But when checking the logfile, I get the following error:

Text File Reader: Source dataset 'P:\Projecten\Nijmegen\Nijmegen BMON 
Ontzorging VSMS 301474\4. FME\SampleJson.txt' does not exist

Posted data (body which is an exact copy of the contents of my textfile-reader)

{
  "type": "SenMLPackPayload",
  "senMLPack": [
  {
  "bn": "urn:dev:com.sensoterra:18000167260",
  "bt": 1532089461
  },   
  {
  "n": "soilmoisture",
  "u": "%vol",
  "v": 11.4,
  "depth": 15
  }
  ]
} 

 

Conclusion/Question

So, to conclude; although I think I followed all the steps, FME-server 
still tries to read the original file-data instead of the body/content 
of the HTML POST.

Is there anyone who know why this is happening?

Some extra information:

 

  • POST is allowed according to the CORS-settings
  • I tried yesterday using FME 2018.0 and today using the brandnew 2018.1
  • I tried changing the original reader to a filereader to see if this makes any difference
  • In the final situation the data is posted to our server by another party. So I'm not able to influence the content of the post. (So I cannot change the content of the json to an workspace-parameter/attribute)

icon

Best answer by siennaatsafe 21 July 2018, 00:34

View original

9 replies

jdh
Rank
Badge +22

I generally just have a text parameter called JSONObject, and use a Creator-> ParameterFetcher ->JSON Transformation rather than a reader.

 

 

Caveats:

In some versions of FME there is a difference in behaviour in regards to escape characters and accessing the JSON parameter directly vs fetching it and accessing it via the resultant attribute.

In some versions of FME there is a difference in behaviour in regards to backslashes between parameter types Text and Text Multiline

K
Badge +3

I generally just have a text parameter called JSONObject, and use a Creator-> ParameterFetcher ->JSON Transformation rather than a reader.

 

 

Caveats:

In some versions of FME there is a difference in behaviour in regards to escape characters and accessing the JSON parameter directly vs fetching it and accessing it via the resultant attribute.

In some versions of FME there is a difference in behaviour in regards to backslashes between parameter types Text and Text Multiline

Thanks for thr fast reply. The downside of this solution is that the body of the post has to contain something like {parameter: jsoncode}

 

 

and since i cannot influence the posted body itself this is not possible as far as I know.

 

 

siennaatsafe
Rank
Badge +9

Hello Koen,

Thank you for your post to the community. I was able to get it to work and I'm hoping that this is the solution you are looking for.

1) I opened up your workspace and ensured that the reader parameter was optional. To do this select SourceDataset_TEXTLINE, then click Edit Value, and then select optional.

2) I added a logger to ensure that the correct data was written in.

3) I uploaded your workspace to the FME Server, using the Job Submitter Service and setting the properties send HTTP Message Body to Reader: SampleJson [TEXTLINE].

4) I went to the FME Server to find the DirectURL for the workspace. To do this go to the Workspace on the FME Server, open the advanced parameters, and find the Direct URL. 2018-07-20-15-20-30.png

5) I set the security settings to allow guests to access my repository. To do this in the security panel of the FME Server, click on users, find guest and double-click to access the settings. 2018-07-20-15-23-26.png Once you are on the guest page, find permissions, find the repository you uploaded to and give guest the permissions to run. 2018-07-20-15-27-26.png

6) Copy the DirectURL into Postman and select the POST method. 2018-07-20-15-28-56.png

Then, click the body section and add in your JSON. 2018-07-20-15-30-01.png

7) Click Send and review!! You should see the data was uploaded to the Server. If you check the job log you should see this worked. 2018-07-20-15-31-45.png

Hopefully, this solution works for you. If not please send me the call you were trying to use.

Also, in two weeks we have a full course on the REST API if you are interested. https://www.safe.com/training/registration/7010c000002WjbFAAS/

https://blog.safe.com/2018/06/new-fme-training-course-build-web-application-rest-api/

takashi
Rank
Userlevel 2
Badge +17

Make sure that the URL syntax is correct as a request of Job Submitter Service in Transformation Services - FME Server Web Services.

I'm afraid that you are going to submit the job via FME Server REST API. Just be aware that the URL syntax of Transformation Services is different from REST API.

K
Badge +3

Hello Koen,

Thank you for your post to the community. I was able to get it to work and I'm hoping that this is the solution you are looking for.

1) I opened up your workspace and ensured that the reader parameter was optional. To do this select SourceDataset_TEXTLINE, then click Edit Value, and then select optional.

2) I added a logger to ensure that the correct data was written in.

3) I uploaded your workspace to the FME Server, using the Job Submitter Service and setting the properties send HTTP Message Body to Reader: SampleJson [TEXTLINE].

4) I went to the FME Server to find the DirectURL for the workspace. To do this go to the Workspace on the FME Server, open the advanced parameters, and find the Direct URL. 2018-07-20-15-20-30.png

5) I set the security settings to allow guests to access my repository. To do this in the security panel of the FME Server, click on users, find guest and double-click to access the settings. 2018-07-20-15-23-26.png Once you are on the guest page, find permissions, find the repository you uploaded to and give guest the permissions to run. 2018-07-20-15-27-26.png

6) Copy the DirectURL into Postman and select the POST method. 2018-07-20-15-28-56.png

Then, click the body section and add in your JSON. 2018-07-20-15-30-01.png

7) Click Send and review!! You should see the data was uploaded to the Server. If you check the job log you should see this worked. 2018-07-20-15-31-45.png

Hopefully, this solution works for you. If not please send me the call you were trying to use.

Also, in two weeks we have a full course on the REST API if you are interested. https://www.safe.com/training/registration/7010c000002WjbFAAS/

https://blog.safe.com/2018/06/new-fme-training-course-build-web-application-rest-api/

Hello Sienna,

 

 

Thanks for your reply and testing. I tried to follow your solution, but I think two things are different than my setup:

 

-I'm trying to use the REST API, and not the directURL transformation services. I think this is also what @takashi means, isnt it?

 

-I have to use a login with token authorization since I want to secure the api.

 

 

So effectively the only thing I changed from my previous situation is making the reader parameter optional and adding a logger. Unfortunately this did not solve the problem yet. Are there any other suggestions?

 

 

Thanks for all the help so far!

 

K
Badge +3

Make sure that the URL syntax is correct as a request of Job Submitter Service in Transformation Services - FME Server Web Services.

I'm afraid that you are going to submit the job via FME Server REST API. Just be aware that the URL syntax of Transformation Services is different from REST API.

Just to be sure if I understand what you mean. When I currently use Postman to post the command to run the workspace, it does the workspace, but gives an error in the log file because it is not able to find the original file. Can this be influenced by the settings/syntax?

 

 

takashi
Rank
Userlevel 2
Badge +17

Make sure that the URL syntax is correct as a request of Job Submitter Service in Transformation Services - FME Server Web Services.

I'm afraid that you are going to submit the job via FME Server REST API. Just be aware that the URL syntax of Transformation Services is different from REST API.

The "Send HTTP Message Body to Reader" property is for Transformation Services in FME Server Web Services. See also here: Register Services. I don't think the property will also be applied to a request via FME Server REST API.

 

You will have to follow the URL syntax for Job Submitter Service if you need to upload the JSON document as the body of POST request.

 

See here to learn about the authentication for FME Server Web Services: Web Services Authentication (HTTP Basic Authentication or Token Authentication).

 

K
Badge +3

Hello Koen,

Thank you for your post to the community. I was able to get it to work and I'm hoping that this is the solution you are looking for.

1) I opened up your workspace and ensured that the reader parameter was optional. To do this select SourceDataset_TEXTLINE, then click Edit Value, and then select optional.

2) I added a logger to ensure that the correct data was written in.

3) I uploaded your workspace to the FME Server, using the Job Submitter Service and setting the properties send HTTP Message Body to Reader: SampleJson [TEXTLINE].

4) I went to the FME Server to find the DirectURL for the workspace. To do this go to the Workspace on the FME Server, open the advanced parameters, and find the Direct URL. 2018-07-20-15-20-30.png

5) I set the security settings to allow guests to access my repository. To do this in the security panel of the FME Server, click on users, find guest and double-click to access the settings. 2018-07-20-15-23-26.png Once you are on the guest page, find permissions, find the repository you uploaded to and give guest the permissions to run. 2018-07-20-15-27-26.png

6) Copy the DirectURL into Postman and select the POST method. 2018-07-20-15-28-56.png

Then, click the body section and add in your JSON. 2018-07-20-15-30-01.png

7) Click Send and review!! You should see the data was uploaded to the Server. If you check the job log you should see this worked. 2018-07-20-15-31-45.png

Hopefully, this solution works for you. If not please send me the call you were trying to use.

Also, in two weeks we have a full course on the REST API if you are interested. https://www.safe.com/training/registration/7010c000002WjbFAAS/

https://blog.safe.com/2018/06/new-fme-training-course-build-web-application-rest-api/

Finally it works now! Thanks for your help Sienna. I was just following the wrong/another path with the REST API with token authorization, while I should use the jobsubmitter service in this situation (as Takashi also mentions). To improve safety I do not use the guest login but instead the basic authentication method.

 

 

Just one question left: it would be great if this can become a part of the REST API too. Is this possible?

 

I subscribed for the course on the REST API, so maybe I can get some answers at that moment:)

 

 

 

siennaatsafe
Rank
Badge +9
Finally it works now! Thanks for your help Sienna. I was just following the wrong/another path with the REST API with token authorization, while I should use the jobsubmitter service in this situation (as Takashi also mentions). To improve safety I do not use the guest login but instead the basic authentication method.

 

 

Just one question left: it would be great if this can become a part of the REST API too. Is this possible?

 

I subscribed for the course on the REST API, so maybe I can get some answers at that moment:)

 

 

 

Hi Koen,

 

I'm happy it's working. We do have the Transact Data Call, https://docs.safe.com/fme/html/FME_REST/apidoc/v3/index.html#!/transformations/post_post_24. It runs through our REST Service which is very similar to the job submitter. However, you may find a Topic may be a better option for you. If you go to create a new topic and go to the advanced settings you will see it lists a POST URL with the token needed. Running the POST URL with the JSON could trigger the workspace to run. Here is the link to the documentation if you would like to explore that option: https://docs.safe.com/fme/html/FME_Server_Documentation/Content/ReferenceManual/FME-Workspace-Subscriber.htm.

 

 

Thanks,

 

Sienna

 

 

Reply


Community Stats

28,699
Posts
109,450
Replies
36,548
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.