Question

How can I test/control an api only processes one feature at a time before sending the next feature into the API?


Badge

I currently use Decelerator but not the best as the time taken to process different features varies, If this was not in place some features don't get processed as too any features are going to the API for it to cope with and therefore returned back with no results. Ideally want to know the feature has come out of the API and then start the next feature.


10 replies

Badge +16

Just wondering: Is there a responce form the API when the processing is done? if so, you could use that to trigger the next feature to be sent to the API.

Badge

Just wondering: Is there a responce form the API when the processing is done? if so, you could use that to trigger the next feature to be sent to the API.

there is a response, I get back my data with additional attributes returned, but not sure how to control the subsequent features in the queue not yet processed (using HTTPCaller).

Userlevel 4
Badge +25

there is a response, I get back my data with additional attributes returned, but not sure how to control the subsequent features in the queue not yet processed (using HTTPCaller).

First solution that springs to mind is to put the HTTPCaller inside a different workspace. Call it for each feature using the WorkspaceRunner transformer. Set it to wait for completion before each run. You just need a way to get the feature into and out of the workspace (probably a FeatureWriter and FeatureReader).

 

 

But I'll try and think of a solution that doesn't involve multiple workspaces.

 

Badge +22
First solution that springs to mind is to put the HTTPCaller inside a different workspace. Call it for each feature using the WorkspaceRunner transformer. Set it to wait for completion before each run. You just need a way to get the feature into and out of the workspace (probably a FeatureWriter and FeatureReader).

 

 

But I'll try and think of a solution that doesn't involve multiple workspaces.

 

That was my first response too (workspace runner).

 

The only other non scripting way I can think of is to aggregate the features with list, and then use a custom transformer to loop through the list. The HTTPCaller would be forked into two streams, one that removes the list (and restores the individual geometry) and gets output, and the other that loops back to a ListIndexer, incrementing the index value.

 

 

Badge

Just wondering: Is there a responce form the API when the processing is done? if so, you could use that to trigger the next feature to be sent to the API.

I did read an example that used the workspace runner but it had the overhead of having to open and close FME, and I was leaning towards looping, is there any example where I could look at building loops (not done those before), that would be close to what I may need?

 

Userlevel 4
Badge +25

The only way I can think to keep this in one workspace is to send a message back to the start of the workspace without which the next feature won't advance (it is stopped by the lack of a FeatureMerger supplier). And the only way I can think of to do that is with a queue such as the Amazon SQS, like this:

Even then I'm not sure if the FeatureMerger will wait for a supplier, or just count it as NotMerged.

Anyone else with any bright ideas?

Userlevel 4
Badge +25
I did read an example that used the workspace runner but it had the overhead of having to open and close FME, and I was leaning towards looping, is there any example where I could look at building loops (not done those before), that would be close to what I may need?

 

The advanced training manual has a helpful section on looping: https://www.gitbook.com/book/safe-software/fme-desktop-advanced-training-2016/details

 

Here you would want to create the loop counter and probably pull out list entries using the ListIndexer pointing to LoopCounter as the item to extract.

 

 

Badge +22

Here's a basic skeleton of how I would set up the loop.

waitforhttp.fmw

Badge +16

there is a response, I get back my data with additional attributes returned, but not sure how to control the subsequent features in the queue not yet processed (using HTTPCaller).

In that case I would build a custom transformer to process each feature individually.

 

The examples supplied can get you started on how to implement such a looping solution.
Badge

Thanks I will have a look at the options suggested

Reply