Skip to main content
Solved

Understanding data streaming ?

  • August 12, 2021
  • 1 reply
  • 202 views

lifalin2016
Contributor
Forum|alt.badge.img+29

Hi list.

(This is not about the new streams feature)

We're trying to utilize FME server for processes, that accept a streaming input and produces a streaming output. I.e. HTTP handlers.

As far as I can tell, you can't initiate such a process via the REST api, as this overrules the output with its own JSON stream. You need to create and use a webhook. Is this correct ?

Even with a webhook it seems, that the streaming input (= POST body) is ignored. Are there some special requirements in the formatting of this input ?

Our input is a custom XML format.

Our output needs to be a CAD file. All examples only name XML and JSON as options, so is this even possible ?

If so, what are the specific requirements to accomplish this ?

Any insights are appreciated.

Cheers.

Best answer by virtualcitymatt

Yes, You will need to use the data steaming service to trigger the workspace e.g.,

http://<host>/fmedatastreaming/<repository>/<workspace>.fmw?<parameters>

When publishing the workspace you need to set up the reader to receive the content of the HTTP Post.

 

The POST body seems to be ignored unless the reader you are using has a normal input file path (i.e., not an attribute). I used a hard coded fake path that pointed to nowhere. e.g., "myinputfile.json".

I have also found issues with Custom Formats not working as expected with receiving a POST body. My workaround was to add in a Textline reader to receive the POST content and then just write the content of the POST to a temp file before reading that temp file with my Custom Reader (as FeatureReader). A bit hacky but seemed to work for me.

 

Another issue I've found is that if the request does not contain a body then the reader will default back to the original input file, if it can't find the default file then it will fail. You can set the workspace to ignore failed readers, however, the logfile still shows an error.

 

I should note that for me I was using the DataDownload Serveice and not the DataStreaming Service but I think that these should function the same way.

View original
Did this help you find an answer to your question?

1 reply

virtualcitymatt
Celebrity
Forum|alt.badge.img+35
  • Celebrity
  • Best Answer
  • August 12, 2021

Yes, You will need to use the data steaming service to trigger the workspace e.g.,

http://<host>/fmedatastreaming/<repository>/<workspace>.fmw?<parameters>

When publishing the workspace you need to set up the reader to receive the content of the HTTP Post.

 

The POST body seems to be ignored unless the reader you are using has a normal input file path (i.e., not an attribute). I used a hard coded fake path that pointed to nowhere. e.g., "myinputfile.json".

I have also found issues with Custom Formats not working as expected with receiving a POST body. My workaround was to add in a Textline reader to receive the POST content and then just write the content of the POST to a temp file before reading that temp file with my Custom Reader (as FeatureReader). A bit hacky but seemed to work for me.

 

Another issue I've found is that if the request does not contain a body then the reader will default back to the original input file, if it can't find the default file then it will fail. You can set the workspace to ignore failed readers, however, the logfile still shows an error.

 

I should note that for me I was using the DataDownload Serveice and not the DataStreaming Service but I think that these should function the same way.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings