Question

Custom transformer to evaluate custom function on dynamic set of attributes

  • 12 September 2016
  • 3 replies
  • 3 views
M
Badge

Hi

 

 

I have a workflow where we are calling a custom arithmetic function (DoubleAsRaw64) to generate a new attribute value based on the old value - this happen in 20+ workplaces on many different attributes.

 

 

We currently do this by using the AttributeManager and:

 

1) Checking if the attribute value has a value

 

2) If it has a value then we do @Evaluate(@DoubleAsRaw64(@Value(NameOfAttribute)))

 

 

I would like to encapsulate this in an custom transformer where I would just have to check of the Attributes on a list - and the the transformer will then apply the DoubleAsRaw64 function on those attributes. This would create a cleaner workplace and minimize the risk for typo etc.

 

 

I have tried to create a super simple custom transformer with one Attribute List as a published parameter Attributes_to_Raw64.

 

And a single ExpressionEvaluator set to overwrite the attributes in $(Attributes_to_Raw64)

 

the Arithmetic Expression is @Evaluate(@DoubleAsRaw64(@CurrentAttribute()))

 

 

This works if I have a single attribute selected in Attributes_to_Raw64 - but as soon as I select more attributes the transformer fails.

 

 

So my questions are:

 

1) How are I implement this so it works with more than one attribute selected to be evaluated per feature?

 

2) How do I best implement a quick null check?

 

3) How do I do this in an efficient way (as there will be millions of features passing though this).

 

 

I would be grateful for any pointers! :)

 

Thank you!

Custom Transformer:

3 replies

takashi
Rank
Userlevel 2
Badge +17

Hi @malteohlers, it doesn't seem that you can pass multiple attribute names to the "Attributes to Overwrite" parameter of the ExpressionEvaluator within a custom transformer through its published parameter. It might be a limitation of the current FME, seems we have to wait for a relevant enhancement unfortunately. Please contact Safe support.

A workaround I can think of is to perform the arithmetic operation with a Python script.

  1. Add an "Attribute List (comma delimited)" type parameter to the custom transformer.
  2. Save the parameter value (comma-separated attribute names) into a temporary attribute. e.g. "_attrlist", with a ParameterFetcher.
  3. Use a PythonCaller to perform the operation for each attribute. The script looks like this.
def processFeature(feature):
    for attr in feature.getAttribute('_attrlist').split(','):
        isNull, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isNull:
            <do something if the attribute is null>
        elif isMissing:
            <do something if the attribute is missing>
        else:
            try:
                oldValue = float(feature.getAttribute(attr))
                newValue = <calculate new value based on the old value>
                feature.setAttribute(attr, newValue) # overwrite
            except:
                <do something if an error occurred>

If you need to use the @DoubleAsRaw64 function to get the new value, maybe the "FMEFeature.performFunction" method can be used to call the function, but I'm not sure if it works fine since I don't know its specification.

Note that this method will fail when a specified attribute name contains a comma. If an attribute name could contain commas, it might be difficult to resolve this with a Python script. In such a case, "Attribute List (space delimited)" type parameter and a Tcl script may be another alternative.

mark2atsafe
Rank
Userlevel 4
Badge +25

Hi @malteohlers, it doesn't seem that you can pass multiple attribute names to the "Attributes to Overwrite" parameter of the ExpressionEvaluator within a custom transformer through its published parameter. It might be a limitation of the current FME, seems we have to wait for a relevant enhancement unfortunately. Please contact Safe support.

A workaround I can think of is to perform the arithmetic operation with a Python script.

  1. Add an "Attribute List (comma delimited)" type parameter to the custom transformer.
  2. Save the parameter value (comma-separated attribute names) into a temporary attribute. e.g. "_attrlist", with a ParameterFetcher.
  3. Use a PythonCaller to perform the operation for each attribute. The script looks like this.
def processFeature(feature):
    for attr in feature.getAttribute('_attrlist').split(','):
        isNull, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isNull:
            <do something if the attribute is null>
        elif isMissing:
            <do something if the attribute is missing>
        else:
            try:
                oldValue = float(feature.getAttribute(attr))
                newValue = <calculate new value based on the old value>
                feature.setAttribute(attr, newValue) # overwrite
            except:
                <do something if an error occurred>

If you need to use the @DoubleAsRaw64 function to get the new value, maybe the "FMEFeature.performFunction" method can be used to call the function, but I'm not sure if it works fine since I don't know its specification.

Note that this method will fail when a specified attribute name contains a comma. If an attribute name could contain commas, it might be difficult to resolve this with a Python script. In such a case, "Attribute List (space delimited)" type parameter and a Tcl script may be another alternative.

Yes, I can confirm that action fails - both inside and outside a custom transformer. I'll file a problem report with the developers and let you know when it is fixed. 

 

mark2atsafe
Rank
Userlevel 4
Badge +25

Hi @malteohlers, it doesn't seem that you can pass multiple attribute names to the "Attributes to Overwrite" parameter of the ExpressionEvaluator within a custom transformer through its published parameter. It might be a limitation of the current FME, seems we have to wait for a relevant enhancement unfortunately. Please contact Safe support.

A workaround I can think of is to perform the arithmetic operation with a Python script.

  1. Add an "Attribute List (comma delimited)" type parameter to the custom transformer.
  2. Save the parameter value (comma-separated attribute names) into a temporary attribute. e.g. "_attrlist", with a ParameterFetcher.
  3. Use a PythonCaller to perform the operation for each attribute. The script looks like this.
def processFeature(feature):
    for attr in feature.getAttribute('_attrlist').split(','):
        isNull, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isNull:
            <do something if the attribute is null>
        elif isMissing:
            <do something if the attribute is missing>
        else:
            try:
                oldValue = float(feature.getAttribute(attr))
                newValue = <calculate new value based on the old value>
                feature.setAttribute(attr, newValue) # overwrite
            except:
                <do something if an error occurred>

If you need to use the @DoubleAsRaw64 function to get the new value, maybe the "FMEFeature.performFunction" method can be used to call the function, but I'm not sure if it works fine since I don't know its specification.

Note that this method will fail when a specified attribute name contains a comma. If an attribute name could contain commas, it might be difficult to resolve this with a Python script. In such a case, "Attribute List (space delimited)" type parameter and a Tcl script may be another alternative.

Filed as PR#72231

 

 

Reply


Community Stats

28,729
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.