Skip to main content

Everything is in the title, I have several operations on a single input dataset branching through several levels of operations. Ultimately I have some mongodb writers and python callers loading transformed data.

 

I want to be sure that one python caller won't start before another one has ended (the first one will create documents, and the second one pushes elements into arrays within those documents).

Place a FeatureHolder before the PythonCaller and that will hold all features until the other flow is ended. But be aware that the FeatureHolder is memory intensive as it holds all features in memory (or file).


In addition to the FeatureHolder mentioned by @erik_jan, have a look at the example here: https://knowledge.safe.com/questions/34290/control-the-flow-of-features-in-a-workspace.html

Starting with FME 2016.1 you can also set the connection runtime order by right-clicking on connections where this would apply:


In addition to the FeatureHolder mentioned by @erik_jan, have a look at the example here: https://knowledge.safe.com/questions/34290/control-the-flow-of-features-in-a-workspace.html

Starting with FME 2016.1 you can also set the connection runtime order by right-clicking on connections where this would apply:

I already use the connection runtime order to properly set the order of the creators, but I noticed each creator starts immediately when the previous one end, even if some transformers following the previous one are still running.

 


Place a FeatureHolder before the PythonCaller and that will hold all features until the other flow is ended. But be aware that the FeatureHolder is memory intensive as it holds all features in memory (or file).

hmm... so I might use a logger connected to the first pythonCaller and link it to a feature holder, itself connected to the second pythonCaller to make it wait ? I will give it a try.

 


I already use the connection runtime order to properly set the order of the creators, but I noticed each creator starts immediately when the previous one end, even if some transformers following the previous one are still running.

 

Yes, you need to consider the runtime order in addition to either the FeatureHolder or the Sampler trick mentioned above.

 

It is also possible to use the FeatureMerger for something similar, if you're creative.

Hi @vchalmel

I would like to elaborate on the FeatureHolder with the following suggestion:

Using this implementation will make sure the top branch will be executed completely before the bottom branch is activated since FeatureHolder_2 will wait on the signal from the sampler in the top branch. Because the Sampler only returns the first 0 features, no feature will be added to the bottom branch.


I understand how this works, though it's not intuitive to beginners. I've often thought there should be a transformer equivalent to an electronic transistor. Use one stream of features to control another, without mixing the two streams. Either release individual or groups in one stream based on incoming features from another, or release a whole stream after another has finished, or hit a predetermined threshold.


This problem is bothering me too. Have tried both the "runtime order" and the FeatureHolder approach with no luck at all. Can anyone provide a simple crystalclear .fmw example showing how to set up two database writers, where it is certain that the last one waits for the first one to finish.


This problem is bothering me too. Have tried both the "runtime order" and the FeatureHolder approach with no luck at all. Can anyone provide a simple crystalclear .fmw example showing how to set up two database writers, where it is certain that the last one waits for the first one to finish.

Why not use FeatureWriter transformer, instead of regular writers?

 


In addition to the FeatureHolder mentioned by @erik_jan, have a look at the example here: https://knowledge.safe.com/questions/34290/control-the-flow-of-features-in-a-workspace.html

Starting with FME 2016.1 you can also set the connection runtime order by right-clicking on connections where this would apply:

Set connection run-time order doesn't seem to be available any more in 2018.1. Was it replaced with something else?


Set connection run-time order doesn't seem to be available any more in 2018.1. Was it replaced with something else?

It's still there, here from FME 2019.0.2:

Just be aware that the context menu option is NOT visible unless there is more than one connection going out of the source transformer:


I am using Desktop 2018.1 and I don't see the "Set Connection Runtime Order" option. Assuming it doesn't work for all connections?

The above FeatureHolder=>Sampler pattern works great for me. "Wait for one branch to complete before continuing along the other". What if the first branch never completes? That doesn't have to mean failure. But can I test for that somehow? I'm working on a process that waits for something to complete before sending an email. If that something never happens, can I still trigger an email notification?


I am using Desktop 2018.1 and I don't see the "Set Connection Runtime Order" option. Assuming it doesn't work for all connections?

The above FeatureHolder=>Sampler pattern works great for me. "Wait for one branch to complete before continuing along the other". What if the first branch never completes? That doesn't have to mean failure. But can I test for that somehow? I'm working on a process that waits for something to complete before sending an email. If that something never happens, can I still trigger an email notification?

There is a nofeaturestester on the hub you could try. You an also use an unconditional featuremerger, create a feature to go to the supplier port, if no features enter the requestor you get an output from the unusued suppliers port


Reply