Skip to main content
Question

Is there a way to make sure the Terminator transformer get executed first once some feature is piped to it?

  • October 11, 2017
  • 7 replies
  • 110 views
S
Forum|alt.badge.img

Hi! I have a workspace with two HTTPCaller transformers. Both of their output ports are piped to some logic to generate features and write to database. There is a FeatureHolder between the HttpCallers and the writers, such that nothing will be written before all the HTTP calls are finished (either complete or error).

Both of the HTTPCaller <reject> ports are piped to a Terminator transformer. I want to make sure the workspace will terminate without any feature written to database, if any of the HTTP calls fails. When I test ran the workspace with some failure in HTTP calls, it did what I want. However, because I couldn't find instruction to help me make sure Terminator will always be executed first, because conceptually when a rejected feature is travelling towards the Terminator, another feature may come out of FeatureHolder and race towards the writer.

Any way I can make sure that?

Thank you!

7 replies

todd_davis
Influencer
Forum|alt.badge.img+22
  • todd_davisInfluencer
  • Influencer
  • October 12, 2017

It might be best to post a screenshot because its hard to get a true understanding of the actual configurations?

But if you have two httpcallers, and go out to one featureholder and terminator(s), the termination will occur before a feature gets released from the featureholder. This is the easiest.

If they are instead two httpcallers and going to two seperate featureholder then you run the risk oof one featureholder letting go. But there are numerous options that could be used in this instance, such as making sure the writer order to the database was set after something else, that would write only when things are correct. Or instead of the featureholder, you could use the featuremerger with something uncondtional (e.g. 1= 1), passing the httpcaller output to the requestor and both the rejected ports from the httpcallers to the suppliers. If a record merged you would terminate, while not merged could write to the database

S
Forum|alt.badge.img
  • sui_huang
  • Author
  • October 12, 2017
todd_davis wrote:

It might be best to post a screenshot because its hard to get a true understanding of the actual configurations?

But if you have two httpcallers, and go out to one featureholder and terminator(s), the termination will occur before a feature gets released from the featureholder. This is the easiest.

If they are instead two httpcallers and going to two seperate featureholder then you run the risk oof one featureholder letting go. But there are numerous options that could be used in this instance, such as making sure the writer order to the database was set after something else, that would write only when things are correct. Or instead of the featureholder, you could use the featuremerger with something uncondtional (e.g. 1= 1), passing the httpcaller output to the requestor and both the rejected ports from the httpcallers to the suppliers. If a record merged you would terminate, while not merged could write to the database

Hi @todd_davis fortunately I let both HTTPCallers share one single FeatureHolder and one single Terminator, so seems the termination will always occur first. Do you know whether there is some FME help/manual document that I can use to justify this directly or indirectly? Thank you!

 

 

Below is a diagram of the configuration.

 

 

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • October 12, 2017
sui_huang wrote:
Hi @todd_davis fortunately I let both HTTPCallers share one single FeatureHolder and one single Terminator, so seems the termination will always occur first. Do you know whether there is some FME help/manual document that I can use to justify this directly or indirectly? Thank you!

 

 

Below is a diagram of the configuration.

 

 

The FeatureHolder's documentation should say that it only releases features when every transformer that is upstream (ahead of it and connected, directly or indirectly, to it) is completely done its work. FME works to push every feature as far as it can before going back and trying to get the next one to work on. So, if any HTTPCaller should reject a feature, that would happen *before* the HTTPCaller would tell any downstream Transformers that it is done its work. And since each transformer pushes any feature it emits as far as it can before it tries to emit any other, or before it tells anyone downstream it is done, the graph you've drawn will do as you want. Nothing could ever get out of the FeatureHolder unless the HTTPCaller(s) complete without any problem.

 

 

I'll alert the doc team to this and see if we can explain this better somewhere. This is a great example.

 

 

S
Forum|alt.badge.img
  • sui_huang
  • Author
  • October 12, 2017
fmelizard wrote:
The FeatureHolder's documentation should say that it only releases features when every transformer that is upstream (ahead of it and connected, directly or indirectly, to it) is completely done its work. FME works to push every feature as far as it can before going back and trying to get the next one to work on. So, if any HTTPCaller should reject a feature, that would happen *before* the HTTPCaller would tell any downstream Transformers that it is done its work. And since each transformer pushes any feature it emits as far as it can before it tries to emit any other, or before it tells anyone downstream it is done, the graph you've drawn will do as you want. Nothing could ever get out of the FeatureHolder unless the HTTPCaller(s) complete without any problem.

 

 

I'll alert the doc team to this and see if we can explain this better somewhere. This is a great example.

 

 

Hi @daleatsafe what I have may become the diagram below, the AttributeCreator before the Terminator is not at "upstream" of FeatureHolder. In this situation is there a chance for feature get out of FeatureHolder first?

 

Thank you!

 

 

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • October 12, 2017
sui_huang wrote:
Hi @daleatsafe what I have may become the diagram below, the AttributeCreator before the Terminator is not at "upstream" of FeatureHolder. In this situation is there a chance for feature get out of FeatureHolder first?

 

Thank you!

 

 

Very good question. The AttributeCreator is NOT a blocking transformer, so it will send its feature through to the Terminator BEFORE it tells the HTTPCaller that it is done. If instead you put a FeatureHolder (or any other blocking transformer) BEFORE the Terminator, than that transformer would tell the HTTPCaller it is done BEFORE the Terminator got the feature. And in that case, it is undefined whether the Terminator would get control before the other FeatureHolders let loose.

 

 

Hope this helps.

 

 

S
Forum|alt.badge.img
  • sui_huang
  • Author
  • October 12, 2017
sui_huang wrote:
Hi @daleatsafe what I have may become the diagram below, the AttributeCreator before the Terminator is not at "upstream" of FeatureHolder. In this situation is there a chance for feature get out of FeatureHolder first?

 

Thank you!

 

 

Hi @daleatsafe thank you for your explanation, then I am more confident that the FME workspace should terminate as I expect. Is the "push every feature as far as it can before going back and try to get the next" philosophy of FME already documented somewhere? or this is what you alerted doc team to add?

 

Thank you!

 

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • October 21, 2017
sui_huang wrote:
Hi @daleatsafe thank you for your explanation, then I am more confident that the FME workspace should terminate as I expect. Is the "push every feature as far as it can before going back and try to get the next" philosophy of FME already documented somewhere? or this is what you alerted doc team to add?

 

Thank you!

 

Yes, this is what I asked for from the doc team.

 

 

Reply


Helpful Members This Week

Community Stats

31,341
Posts
118,864
Replies
38,944
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.