Question

Looking for a how-to for sending data to FME Server via REST service.

A
Badge

I'm trying to learn how to leverage FME Server with web services and have a couple of questions. Been following a few tutorials and built a workbench that:

  1. Reads a JSON file using TextLineReader
  2. Massages the data
  3. Writes data to database

Now I would like to publish this as web service... basically I'd like to be able to send JSON to an FME URL and let server pick it up there. I know you it's possible to run a workbench from a URL. But every tutorial I've tried to follow to do this has led me nowhere. I always end up unable to publish to Data Streaming service because there "Nothing To Select".

Would appreciate any guidance as to what tutorial might work for me. In the meantime, I will dig into the documentation here:

https://s3.amazonaws.com/gitbook/Server-REST-API-2019/index.html

 

11 replies

redgeographics
Rank
Userlevel 5
Badge +25

There is the FME Server REST API training, which would be a good place to start.

But maybe it's a good idea to look into your specific issue. How have you set up your workspace?

A
Badge

Thanks for you reply and sorry about the tardiness with mine.

Yes, I need to review the API training. That's on my Jan 2021 docket. Although I guess there is a difference between the FME API and just kicking off a workspace via URL, or maybe overlap at least? I'm familiar with calling/kicking off a process via URL but I haven't really looked into passing something back form the workspace to the caller.

 

Also, I'm in the middle of migrating from 2018, which probably means I need to get a better handle on 2020 before trying to do this right now. Right of the bat, I notice that when I publish to 2020, I don't get the "Direct URL Example" section in my workspace, which I assumed was the place to see the URL for the web service to call. Also, I found it confusing that in one of the how-to pages I reviewed I was instructed to remove the published parameters for my source when I expected that to be the starting point off passing parameters into said URL. Anyway, as you can tell, best thing would be to start with a simple example.

 

If anyone has any good examples/tutorials for calling a workspace from Javascript, I'd love to hear about it.

A
Badge

Thanks for you reply and sorry about the tardiness with mine.

Yes, I need to review the API training. That's on my Jan 2021 docket. Although I guess there is a difference between the FME API and just kicking off a workspace via URL, or maybe overlap at least? I'm familiar with calling/kicking off a process via URL but I haven't really looked into passing something back form the workspace to the caller.

 

Also, I'm in the middle of migrating from 2018, which probably means I need to get a better handle on 2020 before trying to do this right now. Right of the bat, I notice that when I publish to 2020, I don't get the "Direct URL Example" section in my workspace, which I assumed was the place to see the URL for the web service to call.  Also, I found it confusing that in one of the how-to pages I reviewed I was instructed to remove the published parameters for my source when I expected that to be the starting point off passing parameters into said URL. Anyway, as you can tell, best thing would be to start with a simple example. 

 

If anyone has any good examples/tutorials for calling a workspace from Javascript, I'd love to hear about it.

Finally getting back to this after numerous distractions. Got a couple of questions:

1) Created a service that reads a JSON file and write data to RDBMS. I've published this to server, created a topic that takes JSON input and posts that to the service. I can test that and it works, too. Great!

 

2) Created a 'restapi' user with 'fmeguest' role in FME Server based on tutorial here, and after granting permissions detailed in the article, this account can run the workspace. Don't seem to be able to give this new account access to automations/topics to test whether posting to topic works as well. QUESTION: How do I do that, what are the permissions needed?

 

3) Trying to use an FME Javascript sample to call service from the web. This is based on the code provided here. All I changed were these lines:

<form id="example-form">
		<input id="repository-name" type="hidden" value="DEVELOPMENT" name="repository" />
		<input id="workspace-name" type="hidden" value="Testing_WebService.fmw" name="workspace" />
	</form>

my service info:

FMEServer.init({
	server : "http://<my-server-name>",
	token : "my-restapi-account-token-gobbledeegook"
});

Also, commented out:

// FMEServer.generateFormItems( "example-form", json );

since I don't need an interface. A button is enough to just test this. So my interface ends up looking like this:

But the error I get in browser is a "403" and clicking the button results in the  error in the screenshot. QUESTION: What other permissions are needed for the account to post to that service?

4) Finally, I'm looking for examples of how to post to a topic from Javascript? Ultimately, that's where I'm going with this - building functionality to call FME services from the web. I have not exhausted the documentation pages. But if there is a working example I can play with, that would be great. Question: How do I post JSON to a FME Server topic using FME Javascript API?

 

Thanks for any suggestions or feedback in response to any of the above!

J
Badge +2

Hi @agelfert​ 

 

Here's some resources I found that I think might help you:

Using FME Server webhook (what used to be direct url)

  • Search for webhook url on this documentation page. It also has links to permissions information for tokens. You'll need permissions to the job service you're trying to run, the repository, any data folders you're using, etc (the guest account you may still be using has very limited permissions because then a non-auth users doesn't have the ability to cause issues. Tokens are better)
  • Here's an example I made using Google BigQuery that posts JSON to a webhook url
  • This is an example with ArcGIS Survey 123 using webhooks to run a workspace in FME Server
  • Webhooks and FME Server

 

Using the REST API

 

Javascript example

A
Badge

Hi @agelfert​ 

 

Here's some resources I found that I think might help you:

Using FME Server webhook (what used to be direct url)

  • Search for webhook url on this documentation page. It also has links to permissions information for tokens. You'll need permissions to the job service you're trying to run, the repository, any data folders you're using, etc (the guest account you may still be using has very limited permissions because then a non-auth users doesn't have the ability to cause issues. Tokens are better)
  • Here's an example I made using Google BigQuery that posts JSON to a webhook url
  • This is an example with ArcGIS Survey 123 using webhooks to run a workspace in FME Server
  • Webhooks and FME Server

 

Using the REST API

 

Javascript example

hey @jlutherthomas​ ,

Thanks for posting those links - I will look them over carefully. Much appreciated!

A
Badge

@jlutherthomas​ 

Okay, that was tremendously helpful. After creating a webhook for my workspace, I've managed to use the below method found here:

FMEServer.customRequest( webhookUrl, requestType, showResults, parameters, contentType);

Using the webhook URI, requestType = "POST", the same callback function and parameters as before, FME Server will actually accept the request... but then the workspace fails. It looks to be related to encoding or contentType?

53	2021-2-23 12:03:18 | JSONFlattener (JSONQueryFactory): Unexpected character: 'o'. Expected a JSON value: number, string, boolean value, object or array
54	2021-2-23 12:03:18 | The below feature caused the translation to be terminated
55	2021-2-23 12:03:18 | Storing feature(s) to FME feature store file `E:\Safe Software\FME Server\resources\logs\engine\current\jobs\0\job_164_log.ffs'
56	2021-2-23 12:03:18 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
57	2021-2-23 12:03:18 | Feature Type: `FEATURE_TYPE'
58	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.column' has value `2'
59	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.line' has value `1'
60	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.message' has value `Unexpected character: 'o'. Expected a JSON value: number, string, boolean value, object or array'
61	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.severity' has value `invalid value'

I've tried both ''application/json' for contentType and not providing  any info. How do I fix this so that FME will swallow my JSON?

J
Badge +2

@jlutherthomas​ 

Okay, that was tremendously helpful. After creating a webhook for my workspace, I've managed to use the below method found here:

FMEServer.customRequest( webhookUrl, requestType, showResults, parameters, contentType);

Using the webhook URI, requestType = "POST", the same callback function and parameters as before, FME Server will actually accept the request... but then the workspace fails. It looks to be related to encoding or contentType?

53	2021-2-23 12:03:18 | JSONFlattener (JSONQueryFactory): Unexpected character: 'o'. Expected a JSON value: number, string, boolean value, object or array
54	2021-2-23 12:03:18 | The below feature caused the translation to be terminated
55	2021-2-23 12:03:18 | Storing feature(s) to FME feature store file `E:\Safe Software\FME Server\resources\logs\engine\current\jobs\0\job_164_log.ffs'
56	2021-2-23 12:03:18 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
57	2021-2-23 12:03:18 | Feature Type: `FEATURE_TYPE'
58	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.column' has value `2'
59	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.line' has value `1'
60	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.message' has value `Unexpected character: 'o'. Expected a JSON value: number, string, boolean value, object or array'
61	2021-2-23 12:03:18 | Attribute(encoded: UTF-8) : `_json_error{0}.severity' has value `invalid value'

I've tried both ''application/json' for contentType and not providing  any info. How do I fix this so that FME will swallow my JSON?

@agelfert​ 

Can you share a sample of the JSON you're sending to FME?

A
Badge

Sure... my original workbench in Desktop reads a text file that looks like this:

 dummyJSON 

Really just doing a PoC here for my own understanding. Thus the silly names. The text file reader feeds into a JSON Flattener to only pass through desired attributes:

flattenerIn my Javascript code, I do this:

var myJSON = {"UserName": "Johnny Malone",
	"Process": "the Super Riveting Process"
};
 
var parameters = {
        "SourceDataset_TEXTLINE" : JSON.stringify(myJSON )
};
                
FMEServer.customRequest( webhookUrl, requestType, showResults, parameters,'application/json');

In FME Server, my webhook shows those two JSON keys:

webhookKeysThanks for taking a look at this!

A
Badge

@agelfert​ 

Can you share a sample of the JSON you're sending to FME?

@jlutherthomas​ 

Any more suggestions on this? I will continue poking around but stumbling in the dark right now.

A
Badge

Problem Solved.

Instead of doing this:

var jsonBody = { 
     "key1": "value1",
     "key2": "value2"
};
 
var parameters = {
"SourceDataset" : JSON.stringify(jsonBody),
};
    
FMEServer.customRequest( webhookUrl, requestType, showResults,  parameters,'application/json');

I need to be doing this:

var jsonBody = { 
     "key1": "value1",
     "key2": "value2"
};
 
FMEServer.customRequest( webhookUrl, requestType, showResults,  JSON.stringify(jsonBody),'application/json');

The webhook is only expecting those two keys and not the "SourceDataSet" key that the workspace is expecting.

 

Now, I just need to figure out why this whole process runs twice when triggered! 😁  

J
Badge +2

Problem Solved.

Instead of doing this:

var jsonBody = { 
     "key1": "value1",
     "key2": "value2"
};
 
var parameters = {
"SourceDataset" : JSON.stringify(jsonBody),
};
    
FMEServer.customRequest( webhookUrl, requestType, showResults,  parameters,'application/json');

I need to be doing this:

var jsonBody = { 
     "key1": "value1",
     "key2": "value2"
};
 
FMEServer.customRequest( webhookUrl, requestType, showResults,  JSON.stringify(jsonBody),'application/json');

The webhook is only expecting those two keys and not the "SourceDataSet" key that the workspace is expecting.

 

Now, I just need to figure out why this whole process runs twice when triggered! 😁  

Glad you got it working!

Reply


Community Stats

28,669
Posts
109,357
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.