Skip to main content
Question

Python caller input feature replace with new feature


notmyname
Participant
Forum|alt.badge.img

Hello,

This question is primarily caused by this being my first python class implementation in FME. 

The basic question is? How do I output a new set of features while not outputting the original ones?

 

Details below:

  • I am reading in a set of features to a python caller -class-
  • appending them to a dictionary which in turn is converted to a pandas dataframe for analysis on the set of data
  • Issue: I have been having trouble outputting the new data in the way I want to:
    • What I wanted to do was "drop" the original input fme features and create a new set of features with the new values based on the dataframe. I wanted to do this as I thought it would be quicker than matching the new values to the original input feature
  • The issue is shown in the last loop of the code where newFeature = fmeobjects.FMEFeature() is used to create a new feature and assign attribute values, but in the end I am calling self.pyoutput(newFeature) which I believe is referring to the object created from the input features
  • How can I output new features without referring to the original data?

 

  • The current output looks like below, I know I could just remove the old unnecessary columns after the python caller runs, but I also know I am doing this wrong and would like to correct my mistake.
  •  

import fmeobjects
import pandas as pd


class vAnalys(object):
    def __init__(self):
        #self is convention referring to object created from class (input fme feature)

        #create empty dictionary to store values
        self.data = {"Loc":[], 
            "Value":[]} 
        
    def input(self,feature):
        #input is called for each fme feature, so instead of looping a list just append the desired values to the dictionary

        self.data['Loc'].append(feature.getAttribute('Location'))
        self.data['Value'].append(feature.getAttribute('Measurement'))
        
    def close(self):
        #close is not ran for each feature so the processing on the dictionary/dataframe happens here

        #convert the dictionary to pandas dataframe           
        df= pd.DataFrame(self.data)
        #complete interpolation
        dfInterp=df.interpolate(method='linear', limit_direction ='both',limit=10)
        #****
        #Complete analysis on dataframe
        #****

        #Iterate through each row of the dataframe
        #creating a new FMEFeature
        #Setting the values to the new values
        for index, row in dfInterp.iterrows():
           #print(index, row['Loc'], row['Value'])
           newFeature = fmeobjects.FMEFeature()
           newFeature.setAttribute('Locat',row['Loc'])
           newFeature.setAttribute('InterpValue',row['Value'])
           #this is referring to the original object created by the class
           self.pyoutput(newFeature)

9 replies

david_r
Evangelist
  • April 26, 2019

I'm not quite sure I understand the issue.

The loop in your close method does indeed create one new, empty feature for each row in dfInterp. The output features only contain the two attributes "Locat" and "InterpValue", no geometry or other attributes. The input features are not part of the output.

Isn't that what you wanted, or am I misunderstanding something?

 


notmyname
Participant
Forum|alt.badge.img
  • Author
  • Participant
  • April 26, 2019
david_r wrote:

I'm not quite sure I understand the issue.

The loop in your close method does indeed create one new, empty feature for each row in dfInterp. The output features only contain the two attributes "Locat" and "InterpValue", no geometry or other attributes. The input features are not part of the output.

Isn't that what you wanted, or am I misunderstanding something?

 

@david_r thank you for the response. I realized a picture did not post that would help clarify. The input has 3 attributes visible in the image below. As I am calling fmeobjects.FMEFeature() and assigning 2 attributes to the newFeature to be output I was expecting the output to have only two attributes. However, it has 5, the 3 original, and the 2 new ones. I assumed I was misusing some aspect of fmeObjects and that is what I was looking for clarity on.

This example may seem overly specific but it will be useful for anyone trying to use PANDAS in the future.

 

I appreciate you reviewing the post.

 


david_r
Evangelist
  • April 26, 2019

Could you perhaps mean that the attributes visible in FME Workbench does not reflect the Python script? If so, you need to manually configure which attributes are visible ("exposed") after the PythonCaller, using the bottom section of the PythonCaller settings:

This is necessary because FME Workbench cannot, by itself, interpret the Python script to decide which attribute is introduced or removed during the script execution.


david_r
Evangelist
  • April 26, 2019
notmyname wrote:

@david_r thank you for the response. I realized a picture did not post that would help clarify. The input has 3 attributes visible in the image below. As I am calling fmeobjects.FMEFeature() and assigning 2 attributes to the newFeature to be output I was expecting the output to have only two attributes. However, it has 5, the 3 original, and the 2 new ones. I assumed I was misusing some aspect of fmeObjects and that is what I was looking for clarity on.

This example may seem overly specific but it will be useful for anyone trying to use PANDAS in the future.

 

I appreciate you reviewing the post.

 

As I mentioned in the second post, FME doesn't know what's going on in your Python script, so you'll have to explicitely tell it that these three attributes not longer exist on the output schema (as you can see on your screen dump, the attribute values are not present on the output features).

Try adding these three attributes to "Attributes to hide" in the PythonCaller:


notmyname
Participant
Forum|alt.badge.img
  • Author
  • Participant
  • April 26, 2019
david_r wrote:

As I mentioned in the second post, FME doesn't know what's going on in your Python script, so you'll have to explicitely tell it that these three attributes not longer exist on the output schema (as you can see on your screen dump, the attribute values are not present on the output features).

Try adding these three attributes to "Attributes to hide" in the PythonCaller:

Thank You, After your second post I saw the "attributes to hide" setting. So I guess FME just automatically assigns the input attributes to the output feature, -even if it is a completely new feature- unless explicitly told not to. I was concerned I was misusing fmeObjects and could cause in issue of collision between input and output features.

Thanks!


ebygomm
Influencer
Forum|alt.badge.img+32
  • Influencer
  • April 26, 2019
notmyname wrote:

Thank You, After your second post I saw the "attributes to hide" setting. So I guess FME just automatically assigns the input attributes to the output feature, -even if it is a completely new feature- unless explicitly told not to. I was concerned I was misusing fmeObjects and could cause in issue of collision between input and output features.

Thanks!

The attributes are not really there, just the names visible on the canvas. They've not actually been assigned to the features. You can test this by using a pythoncaller on your output features - feature.getAllAttributeNames() will only give you the attributes that you created on the feature.


david_r
Evangelist
  • April 26, 2019
notmyname wrote:

Thank You, After your second post I saw the "attributes to hide" setting. So I guess FME just automatically assigns the input attributes to the output feature, -even if it is a completely new feature- unless explicitly told not to. I was concerned I was misusing fmeObjects and could cause in issue of collision between input and output features.

Thanks!

@egomm is right, FME distinguishes between the feature schema and the actual attribute values. What you saw in the Inspector was the schema definition, the values themselves were gone (and therefore displayed as null).


notmyname
Participant
Forum|alt.badge.img
  • Author
  • Participant
  • April 26, 2019
david_r wrote:

@egomm is right, FME distinguishes between the feature schema and the actual attribute values. What you saw in the Inspector was the schema definition, the values themselves were gone (and therefore displayed as null).

Thanks for all the information. So technically it sounds like the newFeature = fmeobjects.FMEFeature() inherits the schema definition from class instance of the input object


david_r
Evangelist
  • April 26, 2019
notmyname wrote:

Thanks for all the information. So technically it sounds like the newFeature = fmeobjects.FMEFeature() inherits the schema definition from class instance of the input object

Not quite, the feature object doesn't change, it's simply FME assuming that the feature schema doesn't change between the input and output feature, unless you specifically tell it.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings