Solved

How do you initiate a Job Submitter Service via HTTP POST?

R
Badge

Hi all

I'm trying to test a workspace I've published in FME Cloud as a Job Submitter Service using an HTTP POST to kick it off. I think I’m missing something when trying to deliver the POST.

In my workspace I have a Textfile reader to read the JSON delivered in the POST body. All is set up as described in the Safe article 'Send Source Data to FME Server with HTTP POST'.

(https://knowledge.safe.com/articles/1295/send-source-data-to-fme-server-with-http-post.html)

This is what I’m trying to deliver:

{

"parameters":

{

"xmin":"148.88672",

"xmax":"149.61182",

"ymin":"-29.62122",

"ymax":"-29.11378",

"email":"john.smith@email.com"

},

"available_data":[

{

"source":"NSW",

"downloadables":{

"DEMs":{

"1 Metre":[

{

"file_url":"https://s3-ap-southeast-2.amazonaws.com/nsw/Bunarba_1m.zip"

}

]

}

}

},

{

"source" : "Geoscience Australia",

"downloadables" : {

"DEMs" : {

"1 Second" : [

{

"file_url" : "https://s3-ap-southeast-2.amazonaws.com/ga/144fdd4fa6.zip"

}

],

"5 Metre" : [

{

"file_url" : "https://elvis.ga.gov.au.s3.amazonaws.com/ga/adelongtown0112.tif"

}

]

}

}

}

]

}

Originally we were going to include all the json elements in the url but reconsidered due to the potential length of the encoded json.

'm trying to use Postman (https://www.getpostman.com/) to send the JSON to the FME Cloud service.

For the headers I've set 'Accept' and 'Content-Type' to 'application/json' . The jobs get started in FME Cloud but the content in the POST body doesn’t seem to be delivered so the FME reader fails because it receives no content.

The status return in Postman is: 422 Unprocessable Entity.

When I send the POST, the workspace runs but fails with the error:

Text File Reader: Source dataset 'C:/ELVIS/FSDF/POST_example.txt' does not exist

When I published the workspace I set the 'Send HTTP Message Body to Reader' Service Property to my text file reader so it should use the POST body as the input instead of the path defined in the reader. Doesn't seem to be working though so I thought that maybe the json in the message body isn't getting through.

Any thoughts?

Update:

I've been trying to test a simple workspace to try and get the JSON transferred from the POST to the workspace (see posttest.fmw and post-example.txt - the json I'm trying to deliver).

I no longer think it's a problem with the workspace as I've tested the posttest service via another workspace using an HTTPCaller to execute the POST and all works as expected. I see the contents of the POST body in the log file of the test workspace.

So, it must be a settings issue or something else I'm doing wrong with Postman. Maybe the proxy?

icon

Best answer by rkay 2 November 2016, 23:36

View original
1 Attachment

10 replies

david_r
Rank
Userlevel 4

Consider using the REST API rather than the jobsubmitter, example here: https://knowledge.safe.com/questions/35070/fme-job-submitter-token.html

itay
Rank
Badge +16

Hi @rkay, did you upload the text file used by the reader with the workspace? Or placed it in the resources area of your instance and adapted the location of it in the workspace?

This is typical local vs server/cloud confusion, since your workspace is still trying to read a local text file on your cloud location and cannot find it.

I would also recommend using the API for posting.

L
Badge

I created a simple workspace and published it to the jobsubmitter service and used postman successfully.

0684Q00000ArL4BQAV.png

0684Q00000ArLCbQAN.png

0684Q00000ArKr5QAF.png

We can see that FME server created a file with the posted json and point to it for the TEXTLINE_1_DATASET.

File 24610561524785972_sample.txt content as expected:

{
"Test":"Data"
}

My guess is that your content type must be raw.

Regards,

Larry

itay
Rank
Badge +16

I created a simple workspace and published it to the jobsubmitter service and used postman successfully.

0684Q00000ArL4BQAV.png

0684Q00000ArLCbQAN.png

0684Q00000ArKr5QAF.png

We can see that FME server created a file with the posted json and point to it for the TEXTLINE_1_DATASET.

File 24610561524785972_sample.txt content as expected:

{
"Test":"Data"
}

My guess is that your content type must be raw.

Regards,

Larry

I still think that the workspace is trying to read a local txt file  (C:/ELVIS/FSDF/POST_example.txt) and not one which is created on the fly as your example shows

 

 

L
Badge
I still think that the workspace is trying to read a local txt file (C:/ELVIS/FSDF/POST_example.txt) and not one which is created on the fly as your example shows

 

 

I agree.

 

If the workspace has been published with local test path as the source text file and the content type is invalid, the job does not pick the posted json correctly and the default location is used thus the error.

 

I can repro the issue if I use multipart/form-data as content type. I can't repro if I use application/json as content type.

 

So finally, maybe the source dataset has not been selected correctly when the workspace has been published to the job submitter service.

 

I can repro the issue if I don't select a source dataset at that point.

 

 

R
Badge
I agree.

 

If the workspace has been published with local test path as the source text file and the content type is invalid, the job does not pick the posted json correctly and the default location is used thus the error.

 

I can repro the issue if I use multipart/form-data as content type. I can't repro if I use application/json as content type.

 

So finally, maybe the source dataset has not been selected correctly when the workspace has been published to the job submitter service.

 

I can repro the issue if I don't select a source dataset at that point.

 

 

Hi larry,

 

I've tested your suggestions and tried to replicate with a simple workspace, as you did but I get the same errors as before. If I publish the workspace and upload the local text file along with it, when the job is run it always reads the uploaded file and runs successfully. If I don't upload the file I get and Unprocessable Content error in Postman and the FME Cloud job fails because it can't find the file path defined in the text file reader. I'm always publishing with these service properties:

 

With these properties it should create the temp txt file containing the the POST body and read that shouldn't it?

 

 

I'm using FME Cloud version: FME Server 2016.1.0.1 - Build 16494 - linux-x64

 

and desktop version: 2016.1.0.0 (20160419 - Build 16492 - WIN32

 

 

L
Badge
Hi larry,

 

I've tested your suggestions and tried to replicate with a simple workspace, as you did but I get the same errors as before. If I publish the workspace and upload the local text file along with it, when the job is run it always reads the uploaded file and runs successfully. If I don't upload the file I get and Unprocessable Content error in Postman and the FME Cloud job fails because it can't find the file path defined in the text file reader. I'm always publishing with these service properties:

 

With these properties it should create the temp txt file containing the the POST body and read that shouldn't it?

 

 

I'm using FME Cloud version: FME Server 2016.1.0.1 - Build 16494 - linux-x64

 

and desktop version: 2016.1.0.0 (20160419 - Build 16492 - WIN32

 

 

Hi,

 

 

I tested successfully with cloud build 16609 and desktop 16489.

 

 

Can you attach your test workspace and the json you are trying to post to this question please?

 

 

R
Badge

Ok, I've figured out what the problem was. The method I was using was fine - both workspace and with Postman. It was the CORS configuration in FME Cloud that was causing the problem. All I had to do was add POST to the 'Allowed Methods' under Manage -> Administration -> CORS.

Thanks very much for your help with this one larry, itay and david_r.

K
Badge

I tried all the steps described above (and also checked if POST is allowed), but still I get the same error as the original question: so instead of reading the json body it tries to read the original text file. Any other ideas? I use FME server 2018 and Postman for testing.

K
Badge

I tried all the steps described above (and also checked if POST is allowed), but still I get the same error as the original question: so instead of reading the json body it tries to read the original text file. Any other ideas? I use FME server 2018 and Postman for testing.

I just installed fme 2018.1 and fme-server 2018.1 to see if this would fix the issue. But unfortunately it didn't. So if someone is able to help that would be great.

 

 

Why I hoped 2018.1 would offer a solution:

 

FME 2018.1 release blog

 

[...]

 

The FeatureReader and FeatureWriter transformers can now be used as part of FME Server authoring workflows when registering a workspace with a service, like Job Submitter or Data Download.

 

In the options for ‘Send HTTP Message Body to Reader’ you can now select a FeatureReader input

 

Reply


Community Stats

28,670
Posts
109,367
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.