Solved

Executing Variable Setter and Variable Retriever in the right order

  • 6 September 2017
  • 9 replies
  • 16 views
jochgem
Rank
Badge +4

Hi there!

Working with FME Desktop 16.1.2.1 in a rather complex FME Workspace I decided recently to reorganize the FME Workspace such that it would become a little less complex to maintain the Workspace.

Certain attribute values, that were used in multiple occasions in different parts within the Workspace, were before distributed based on FeatureMergers based on common attributes and attribute values (tempid = 1 for all incoming features) and making use of AttributeKeepers to keep the amount of data processing small and let only those attributes through that are actually needed. In that case the order of adding attributes to features is defined and this works splendid (but makes the FME Workspace perhaps a little more complex than needed and less elegant to read).

In order to simplify the Workspace I decided to use VariableSetters and VariableRetrievers throughout the Workspace such that less FeatureMergers and hence less connections are needed, simplifying the Workspace.

Now I am experiencing some problems. In at least one case a VariableRetriever is read before the Variable is set in a VariableSetter. Apparently certain logic within FME Workbench is missing (at least for FME Desktop 16.1.2.1) to first execute the VariableSetter and only afterwards any VariableRetrievers. In those cases the attributes are created without the required attribute values, resulting in this case in empty attributes and a problem in the chain of processing.

I am in search for ways to cope with this problem. I came up with the following workarounds:

* I could make use of the FeatureHolder and could be lucky that that would delay executing the VariableRetriever such that the VariableSetter is executed beforehand. Based on trial and error this could in cases work out. The need of multiple FeatureHolders within the Workspace would complicate the matter however as it is unknown beforehand which FeatureHolders are executed first;

* The order of Readers could be changed such that the order in which the processes are executed changes. Within the Workspace mostly FeatureReaders are used however, and as far as I know the order in which these are executed can not be set;

* FeatureMergers based on common attributes and attribute values (tempid = 1 for all features) could define the order in which the VariableRetrievers and VariableSetters are executed. However, this is undermining the goal to simplify the Workspace using less FeatureMergers and is therefore not a real solution.

 

I am looking forward for suggestions to cope with this problem. Meanwhile I partially will have to undo some of the work I did.

 

Best Regards, Jochgem Gunneman

icon

Best answer by david_r 6 September 2017, 10:36

View original

9 replies

david_r
Rank
Userlevel 4

Could you please be a bit more specific?

Have you looked at the documentation for the VariableSetter and VariableRetriever transformers?

jochgem
Rank
Badge +4

Could you please be a bit more specific?

Have you looked at the documentation for the VariableSetter and VariableRetriever transformers?

Sorry. I pushed the submit button too early. I guess you have been reading only the title?

 

 

david_r
Rank
Userlevel 4
Sorry. I pushed the submit button too early. I guess you have been reading only the title?

 

 

Yes, I only saw the title. Much better now :-)
david_r
Rank
Userlevel 4

It's a well known problem how to guarantee a certain order of operations in an FME workspace, but fortunately this was made a lot easier starting with 2016 where you can specify the connection order when you have multiple outputs:

It's a bit hidden away but I've had much luck with it, it's a whole lot more efficient than using FeatureMergers and FeatureHolders to accomplish the same thing. This technique is also very helpful when unblocking transformers that allow setting an expected input order (e.g. Suppliers first).

jochgem
Rank
Badge +4

Thanks! This is exactly what I need. :-)

jochgem
Rank
Badge +4

I was a little too quick in my answer.

The solution David offers is certainly valuable and in some cases a good solution to guarantee a certain order of FME operations. I noticed that in case the connections are coming out of the same output port the solution is certainly valid. But what about if a Transformer has multiple output ports? In a quick search I could not find a way how the order of different output ports can be orchestrated such that a certain order of FME operations is guaranteed.

 

In my case, based on the extension of the filename, a different protocol follows. Using a TestFilter with multiple output ports files are read and information is merged at a later stage. (Using either FeatureMergers or VariableSetters/VariableRetrievers.)

 

Any suggestions are welcome to solve the problem for Transformers with multiple output ports. :-)

takashi
Rank
Userlevel 2
Badge +17

I was a little too quick in my answer.

The solution David offers is certainly valuable and in some cases a good solution to guarantee a certain order of FME operations. I noticed that in case the connections are coming out of the same output port the solution is certainly valid. But what about if a Transformer has multiple output ports? In a quick search I could not find a way how the order of different output ports can be orchestrated such that a certain order of FME operations is guaranteed.

 

In my case, based on the extension of the filename, a different protocol follows. Using a TestFilter with multiple output ports files are read and information is merged at a later stage. (Using either FeatureMergers or VariableSetters/VariableRetrievers.)

 

Any suggestions are welcome to solve the problem for Transformers with multiple output ports. :-)

The TestFilter is a Feature-Based transformer, the order of output features will be equal to the input order.

 

See here about the term "Feature-Based" in FME:

 

How FME processes features and how it affects the translation process

 

 

jochgem
Rank
Badge +4
The TestFilter is a Feature-Based transformer, the order of output features will be equal to the input order.

 

See here about the term "Feature-Based" in FME:

 

How FME processes features and how it affects the translation process

 

 

Hi Takashi,

 

 

Thanks for this suggestion! This brought me to an idea to solve this specific problem.

 

 

In the article a Sorter is mentioned. Based on the filename extension I can give attribute values that can be sorted by the Sorter. After that a certain order of FME operations can be guaranteed (at least in this case).

 

 

Best Regards, Jochgem

 

 

jochgem
Rank
Badge +4
Hi Takashi,

 

 

Thanks for this suggestion! This brought me to an idea to solve this specific problem.

 

 

In the article a Sorter is mentioned. Based on the filename extension I can give attribute values that can be sorted by the Sorter. After that a certain order of FME operations can be guaranteed (at least in this case).

 

 

Best Regards, Jochgem

 

 

Hi all,

 

 

Another update.

 

 

I just tried the solution by implementing an AttrbuteCreator (based on conditional values) and a Sorter in order to get the files in a specific order through the TestFilter.

 

 

Unfortunately it did not work out for this specific problem. Although the features get in the right order through the TestFilter one of the output ports of the TestFilter is followed up by a FeatureReader. I suspect that this Transformer only starts reading the file after other processes have been carried out, resulting in the wrong order of processing of the VariableSetter/VariableRetriever combination.

 

 

Any suggestions are still welcome!

 

Reply


Community Stats

28,702
Posts
109,463
Replies
36,550
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.