Skip to main content
Question

PythonCreator ('close' method) Runtime Order Setting

takashi
Influencer

The PythonCreator has been improved in FME 2017.1. The 'input' method has been added and the user can choose creating features before any reader via the 'input' method, creating features after any reader via the 'close' method.

It's a good enhancement, but I noticed that the runtime order of the Creators on the 'close' method cannot be controlled. That is, PythonCreators don't appear in the Creators Run After Readers table in the Creator Runtime Order dialog.

See also the attached demo:  (FME 2017.1.2)

0684Q00000ArCoLQAV.png

import fmeobjects
class FeatureCreator1(object):
    def __init__(self):
        pass
    def input(self,feature):
        newFeature = fmeobjects.FMEFeature()
        newFeature.setAttribute('_method', 'input')
        self.pyoutput(newFeature)
    def close(self):
        newFeature = fmeobjects.FMEFeature()
        newFeature.setAttribute('_method', 'close')
        self.pyoutput(newFeature)

In addition, if the source dataset was empty (i.e. the reader feature type outputs no features), the order of features output from the PythonCreators would be changed.

0684Q00000ArCyGQAV.png

  • When features were output from the reader: input - input - close -close.
  • When no features from the reader: input - close - input- close.

It seems to still be a room to improve.

9 replies

debbiatsafe
Safer
Forum|alt.badge.img+20
  • debbiatsafeSafer
  • Safer
  • January 20, 2018

Hi @takashi

Thank you for bringing this to our attention. I have filed enhancement request (PR81520) regarding the Creator Runtime Order dialog's inconsistency with PythonCreator methods and defect report (PR81523) for PythonCreator runtime order not honoured if source dataset is empty.

I'll update you when they have been addressed.

Kind regards,

Debbi

takashi
Influencer
  • takashiInfluencer
  • Author
  • January 20, 2018
debbiatsafe wrote:

Hi @takashi

Thank you for bringing this to our attention. I have filed enhancement request (PR81520) regarding the Creator Runtime Order dialog's inconsistency with PythonCreator methods and defect report (PR81523) for PythonCreator runtime order not honoured if source dataset is empty.

I'll update you when they have been addressed.

Kind regards,

Debbi

Hi @DebbiAtSafe, thanks for filing the PRs.

 

Just an idea. In the Creator Runtime Order dialog, it would be ideal if a suffix indicating the method ('input' or 'close') could be added to the name of PythonCreator in the tables. e.g.

 

  • "PythonCreator #input" in the Creators Run Before Readers table
  • "PythonCreator #close" for the Creators Run After Reader table
debbiatsafe
Safer
Forum|alt.badge.img+20
  • debbiatsafeSafer
  • Safer
  • January 26, 2018
takashi wrote:
Hi @DebbiAtSafe, thanks for filing the PRs.

 

Just an idea. In the Creator Runtime Order dialog, it would be ideal if a suffix indicating the method ('input' or 'close') could be added to the name of PythonCreator in the tables. e.g.

 

  • "PythonCreator #input" in the Creators Run Before Readers table
  • "PythonCreator #close" for the Creators Run After Reader table
Hi @takashi

 

You're very welcome. I will add your idea to the relevant PR.

 

 

Kind regards,

 

Debbi

 

debbiatsafe
Safer
Forum|alt.badge.img+20
  • debbiatsafeSafer
  • Safer
  • January 26, 2018

Hi @takashi 

Regarding PR81523 (incorrect runtime order), our developer has looked into it and said that it's behaviour that is expected. I'll try to summarize his thoughts on this.

 

Pipeline: CreationFactory (CF_PC1) -> PythonFactory (PF_PC1) -> CreationFactory (CF_PC2) -> PythonFactory (PF_PC2) -> AttributeExposer/Inspector factories

 

From my understanding, the feature being read is a trigger. It flows through a pipeline where it will trigger the CF_PC1 to create its 'input' feature which will branch to AttributeExposer/Inspector (end of pipeline). The trigger then flows through and will be noticed by the second PythonCaller's CreationFactory. CF_PC2 creates its 'input' feature and goes to end of pipeline. Then the trigger feature will reach the end of pipeline. As there are no more features, the pipeline will begin to shut down.

 

Since CF_PC1 already created their feature, nothing created when it is shut down. PF_PC1 creates its 'close' feature on shutdown and feature is sent to end of pipeline. CF_PC2 shuts down (nothing created because already created earlier). PF_PC2 creates its 'close' feature on shutdown and feature is sent to end of pipeline. Pipeline shut down completely.

However, in the case of no features being read, there is no trigger. So the CreationFactories and PythonCreations act in sequence when shutdown.

 

 

In summary:

 

With some reader input: input, input, <FeaturesRead>, close, close

 

Without reader input: input, close, input, close

 

 

Our developer also suggested the following:

 

Indeed, if the features from the data were to be emitted after a couple of PythonCreators' initial features, I would run the reader input through the PythonCreators sequentially, with code something like:
def __init__(self):
    self.doneInitialFeature = False
def input(self,feature):
    if not self.doneInitialFeature:
       newFeature = fmeobjects.FMEFeature()
       newFeature.setAttribute('_method', 'input')
       self.pyoutput(newFeature)
       self.doneInitialFeature = True
    self.pyoutput(feature)

Kind regards,

Debbi

takashi
Influencer
  • takashiInfluencer
  • Author
  • January 26, 2018
debbiatsafe wrote:

Hi @takashi 

Regarding PR81523 (incorrect runtime order), our developer has looked into it and said that it's behaviour that is expected. I'll try to summarize his thoughts on this.

 

Pipeline: CreationFactory (CF_PC1) -> PythonFactory (PF_PC1) -> CreationFactory (CF_PC2) -> PythonFactory (PF_PC2) -> AttributeExposer/Inspector factories

 

From my understanding, the feature being read is a trigger. It flows through a pipeline where it will trigger the CF_PC1 to create its 'input' feature which will branch to AttributeExposer/Inspector (end of pipeline). The trigger then flows through and will be noticed by the second PythonCaller's CreationFactory. CF_PC2 creates its 'input' feature and goes to end of pipeline. Then the trigger feature will reach the end of pipeline. As there are no more features, the pipeline will begin to shut down.

 

Since CF_PC1 already created their feature, nothing created when it is shut down. PF_PC1 creates its 'close' feature on shutdown and feature is sent to end of pipeline. CF_PC2 shuts down (nothing created because already created earlier). PF_PC2 creates its 'close' feature on shutdown and feature is sent to end of pipeline. Pipeline shut down completely.

However, in the case of no features being read, there is no trigger. So the CreationFactories and PythonCreations act in sequence when shutdown.

 

 

In summary:

 

With some reader input: input, input, <FeaturesRead>, close, close

 

Without reader input: input, close, input, close

 

 

Our developer also suggested the following:

 

Indeed, if the features from the data were to be emitted after a couple of PythonCreators' initial features, I would run the reader input through the PythonCreators sequentially, with code something like:
def __init__(self):
    self.doneInitialFeature = False
def input(self,feature):
    if not self.doneInitialFeature:
       newFeature = fmeobjects.FMEFeature()
       newFeature.setAttribute('_method', 'input')
       self.pyoutput(newFeature)
       self.doneInitialFeature = True
    self.pyoutput(feature)

Kind regards,

Debbi

Hi @DebbiAtSafe, thanks for your explanation. I can understand how the script in a PythonCreator would be executed at runtime, but my point is not here.

 

The "Creator Runtime Order" functionality has been introduced in FME 2016.1. It was a great enhancement, but was not perfect at first. That is, runtime order of some Creator transformers didn't have been consistent with setting in the "Create Runtime Order" dialog. Then, the inconsistencies have been improved, and it seems to have been almost completed in FME 2017.1.

 

Here is a history: Control the Execution Order of "Starter" Transformers

 

However, in FME 2017.1 [Edited], I noticed that the runtime order of the features created in the 'close' method in PythonCreator cannot be controlled through the "Creator Runtime Order" dialog. It's the issue I pointed.

 

It would be ideal if the runtime order on both "input" and "close" methods could be completely, separately controlled through the "Creator Runtime Order" dialog. Even if it would be hard to control them completely because of a limitation on the implementation, at least it should be documented how the runtime order on the PythonCreator would be determined.

 

takashi
Influencer
  • takashiInfluencer
  • Author
  • January 26, 2018
debbiatsafe wrote:

Hi @takashi 

Regarding PR81523 (incorrect runtime order), our developer has looked into it and said that it's behaviour that is expected. I'll try to summarize his thoughts on this.

 

Pipeline: CreationFactory (CF_PC1) -> PythonFactory (PF_PC1) -> CreationFactory (CF_PC2) -> PythonFactory (PF_PC2) -> AttributeExposer/Inspector factories

 

From my understanding, the feature being read is a trigger. It flows through a pipeline where it will trigger the CF_PC1 to create its 'input' feature which will branch to AttributeExposer/Inspector (end of pipeline). The trigger then flows through and will be noticed by the second PythonCaller's CreationFactory. CF_PC2 creates its 'input' feature and goes to end of pipeline. Then the trigger feature will reach the end of pipeline. As there are no more features, the pipeline will begin to shut down.

 

Since CF_PC1 already created their feature, nothing created when it is shut down. PF_PC1 creates its 'close' feature on shutdown and feature is sent to end of pipeline. CF_PC2 shuts down (nothing created because already created earlier). PF_PC2 creates its 'close' feature on shutdown and feature is sent to end of pipeline. Pipeline shut down completely.

However, in the case of no features being read, there is no trigger. So the CreationFactories and PythonCreations act in sequence when shutdown.

 

 

In summary:

 

With some reader input: input, input, <FeaturesRead>, close, close

 

Without reader input: input, close, input, close

 

 

Our developer also suggested the following:

 

Indeed, if the features from the data were to be emitted after a couple of PythonCreators' initial features, I would run the reader input through the PythonCreators sequentially, with code something like:
def __init__(self):
    self.doneInitialFeature = False
def input(self,feature):
    if not self.doneInitialFeature:
       newFeature = fmeobjects.FMEFeature()
       newFeature.setAttribute('_method', 'input')
       self.pyoutput(newFeature)
       self.doneInitialFeature = True
    self.pyoutput(feature)

Kind regards,

Debbi

It would be ideal if the user could control the runtime order of all the Creator transformers with the Creator Runtime Order dialog shown like this.

 

0684Q00000ArMP8QAN.png

 

Looking at this, I still think that the PythonCreators should output features in the order of "input-input-close-close" according to the order set in the "Creator Runtime Order" dialog, even if there were no readers. How do you think?

 

 

david_r
Celebrity
  • david_rCelebrity
  • January 26, 2018
takashi wrote:
It would be ideal if the user could control the runtime order of all the Creator transformers with the Creator Runtime Order dialog shown like this.

 

 

Looking at this, I still think that the PythonCreators should output features in the order of "input-input-close-close" according to the order set in the "Creator Runtime Order" dialog, even if there were no readers. How do you think?

 

 

Yes, I agree, this would be ideal. As more and more transformers support notions such as "suppliers first" etc it becomes increasingly important to be able to have fine-grained control over the feature order.
debbiatsafe
Safer
Forum|alt.badge.img+20
  • debbiatsafeSafer
  • Safer
  • January 27, 2018
debbiatsafe wrote:

Hi @takashi 

Regarding PR81523 (incorrect runtime order), our developer has looked into it and said that it's behaviour that is expected. I'll try to summarize his thoughts on this.

 

Pipeline: CreationFactory (CF_PC1) -> PythonFactory (PF_PC1) -> CreationFactory (CF_PC2) -> PythonFactory (PF_PC2) -> AttributeExposer/Inspector factories

 

From my understanding, the feature being read is a trigger. It flows through a pipeline where it will trigger the CF_PC1 to create its 'input' feature which will branch to AttributeExposer/Inspector (end of pipeline). The trigger then flows through and will be noticed by the second PythonCaller's CreationFactory. CF_PC2 creates its 'input' feature and goes to end of pipeline. Then the trigger feature will reach the end of pipeline. As there are no more features, the pipeline will begin to shut down.

 

Since CF_PC1 already created their feature, nothing created when it is shut down. PF_PC1 creates its 'close' feature on shutdown and feature is sent to end of pipeline. CF_PC2 shuts down (nothing created because already created earlier). PF_PC2 creates its 'close' feature on shutdown and feature is sent to end of pipeline. Pipeline shut down completely.

However, in the case of no features being read, there is no trigger. So the CreationFactories and PythonCreations act in sequence when shutdown.

 

 

In summary:

 

With some reader input: input, input, <FeaturesRead>, close, close

 

Without reader input: input, close, input, close

 

 

Our developer also suggested the following:

 

Indeed, if the features from the data were to be emitted after a couple of PythonCreators' initial features, I would run the reader input through the PythonCreators sequentially, with code something like:
def __init__(self):
    self.doneInitialFeature = False
def input(self,feature):
    if not self.doneInitialFeature:
       newFeature = fmeobjects.FMEFeature()
       newFeature.setAttribute('_method', 'input')
       self.pyoutput(newFeature)
       self.doneInitialFeature = True
    self.pyoutput(feature)

Kind regards,

Debbi

Hi @takashi

 

You're welcome. 

 

I do agree with you and @david_r about being able to control "close" method with the Creator Runtime Dialog would be ideal. This enhancement request is still currently active.

 

My explanation above was an attempt to pass on one of our developer's thoughts regarding the input-input-close-close situation. After reading his explanation, I think I understand why it is happening. 

 

takashi
Influencer
  • takashiInfluencer
  • Author
  • January 27, 2018
debbiatsafe wrote:
Hi @takashi

 

You're welcome.

 

I do agree with you and @david_r about being able to control "close" method with the Creator Runtime Dialog would be ideal. This enhancement request is still currently active.

 

My explanation above was an attempt to pass on one of our developer's thoughts regarding the input-input-close-close situation. After reading his explanation, I think I understand why it is happening.

 

Yes, I can understand why it is happening but cannot agree saying it is conformed to the concept of the "Creator Runtime Order" functionality.

 

Thanks for your consideration.

 

 

Reply


Helpful Members This Week

Community Stats

31,344
Posts
118,879
Replies
38,940
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.