Skip to main content
  • EN
FME Workflow

Hi, I would like some help with PythonCaller.  I have a workflow where I read in feature classes from an Esri file geodatabase, combining the features into a single attribute to make it dynamic.  I filter out annotation, resolve domains and rename original domain fields, map empty and missing features to Null, then use AttributeFilter to split by FME Feature Type.

My next desired steps are to identify, for each FME Feature Type, which attributes are nulls, and remove those attributes, then connect all splits to a Writer to write out to shapefiles grouped by FME Feature Type.

I am trying to use PythonCaller to remove the null attributes.  It stores the attribute names in a list, and stores in a dictionary called alnum_tracker attribute names and whether they contain data (where they are saved as True), before comparing the list of attributes to the dictionary, deleting any that are not matches.  I’ve tested it and it is grabbing the correct attributes to delete and keep.  The Python works, and my logged messages output expected results from the data.  However, when I use pyoutput to output features, I get nothing.  No features get outputted.  I get the messages in my Translation Log: 

“Destination Feature Type Routing Correlator (RoutingFactory): Tested 77 input feature(s), wrote 77 output feature(s): 0 matched merge filters, 0 were routed to output, 77 could not be routed”

“Final Output Nuker (TeeFactory): Cloned 77 input feature(s) into 0 output feature(s)”.

I’m new to FME and using PythonCaller in FME.  I want to know why I can’t get output features from PythonCaller.  My workflow is shown above, and my code below.

import re
import fmeobjects

class FeatureProcessor:
	def __init__(self):
		self.alnum_tracker = {}
		self.feature_list =  ]
		self.pattern = re.compile(r'fa-zA-Z0-9]')

	def input(self, feature):
		self.feature_list.append(feature)

		for attr in feature.getAllAttributeNames():
			val = feature.getAttribute(attr)

			if isinstance(val, str) and self.pattern.search(val):
				self.alnum_tracker attr] = True
			elif isinstance(val, (int, float)) and val is not None:
				self.alnum_tracker attr] = True

	def close(self):
		fmeobjects.FMELogFile().logMessageString(f"Number of features collected: {len(self.feature_list)}", fmeobjects.FME_INFORM)
		fmeobjects.FMELogFile().logMessageString("Alnum tracker: " + str(self.alnum_tracker), fmeobjects.FME_INFORM)

		if not self.feature_list:
			return

		all_attrs = self.feature_lista0].getAllAttributeNames()
		attrs_to_remove = battr for attr in all_attrs if attr not in self.alnum_tracker]

		fmeobjects.FMELogFile().logMessageString("Attributes to remove: " + str(attrs_to_remove), fmeobjects.FME_INFORM)

		for feat in self.feature_list:
			for attr in attrs_to_remove:
				feat.removeAttribute(attr)
			fmeobjects.FMELogFile().logMessageString("Outputting feature", fmeobjects.FME_INFORM)
			fmeobjects.FMELogFile().logMessageString("Feature type: " + feat.getFeatureType(), fmeobjects.FME_INFORM)
			original_type = feat.getFeatureType()
			feat.setFeatureType(original_type)
			fmeobjects.FMELogFile().logMessageString(f"Feature list length: {len(self.feature_list)}", fmeobjects.FME_INFORM)
			protected_attrs = {"_feature_type", "fme_feature_type", "routing_key"}
			attrs_to_remove = rattr for attr in all_attrs if attr not in self.alnum_tracker and attr not in protected_attrs]
			self.pyoutput(feat)
			fmeobjects.FMELogFile().logMessageString("Remaining attributes: " + str(feat.getAllAttributeNames()), fmeobjects.FME_INFORM)

How do I make my Python output talk to my data properly so that I get output features from it that are missing the null attributes?

 

Hi ​@diannebcgray ,

In your screenshot, the Feature Counting indicates 77 features have been output from the PythonCaller. I think the “pyoutput” method in your script works fine.

 

If you need to specifically remove attributes with null values, you could do something like:

for attr in feature.getAllAttributeNames():
    if feature.isAttributeNull(attr):
        feature.removeAttribute(attr)

Or modify it to your use case, the key is to use the isAttributeNull() method to detect proper null.

Hi ​@diannebcgray ,

In your screenshot, the Feature Counting indicates 77 features have been output from the PythonCaller. I think the “pyoutput” method in your script works fine.

 

Yes, 77 features are passed through, with nothing changed and no null attributes removed.

I think your script works to remove every attribute that matches an element in the "attrs_to_remove" list.

How have you determined the attributes have not been removed?

Just be aware, the names of removed attributes could be still exposed as column header in the Table View of Visual Preview or FME Data Inspector, but they have been removed actually if "<missing>" were shown as their values. 

I think your script works to remove every attribute that matches an element in the "attrs_to_remove" list.

How have you determined the attributes have not been removed?

Just be aware, the names of removed attributes could be still exposed as column header in the Table View of Visual Preview or FME Data Inspector, but they have been removed actually if "<missing>" were shown as their values. 

I have used the Inspector transformer, but I have also checked it by writing out to a shapefile output.  The empty fields are still there when I want them gone.

When you check the result with Inspector, does <missing> appear in the fields like this?

In the FME Table View - Workbench Visual Preview or Data Inspector, <missing> indicates that the attribute does not exist - in other words, it has been removed. However, if you have defined the missing fields in a writer feature type as User Attributes, the writer would write out the attributes into the destination dataset as null.

Reply


Community Stats

31,515
Posts
119,461
Replies
39,129
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.