Is there a transformer that dynamically identifies features where all attribute columns are empty? This could be done with a Tester, but that requires the attributes to be known, and a new setup for each dataset. I was curious if it can be done “on the fly”.
You could do this using the PythonCaller and the getAllAttributeNames function - https://docs.safe.com/fme/html/fmepython/api/fmeobjects/_feature/fmeobjects.FMEFeature.getAllAttributeNames.htm
Something like:
import fme
import fmeobjects
class FeatureProcessor(object):
"""Template Class Interface:
When using this class, make sure its name is set as the value of the 'Class to Process Features'
transformer parameter.
"""
def __init__(self):
"""Base constructor for class members."""
pass
def has_support_for(self, support_type: int):
"""This method is called by FME to determine if the PythonCaller supports Bulk mode,
which allows for significant performance gains when processing large numbers of features.
Bulk mode cannot always be supported.
More information available in transformer help.
"""
return support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM
def input(self, feature: fmeobjects.FMEFeature):
"""This method is called for each feature which enters the PythonCaller.
Processed input features can be emitted from this method using self.pyoutput().
If knowledge of all input features is required for processing, then input features should be
cached to a list instance variable and processed using group processing or in the close() method.
"""
emptycount = 0
featcount = 0
for f in feature.getAllAttributeNames():
if f.startswith('fme'):
pass
else:
featcount += 1 #keep count of number of features
if feature.getAttribute(f) == '':
#empty string
emptycount += 1 #increment empty count
else:
#has a vlaue
pass
if emptycount == featcount:
#all attrs empty
feature.setAttribute('AllEmpty',True)
else:
feature.setAttribute('AllEmpty',False)
self.pyoutput(feature)
def close(self):
"""This method is called once all the FME Features have been processed from input()."""
pass
def process_group(self):
"""This method is called by FME for each group when group processing mode is enabled.
This implementation should reset any instance variables used for the next group.
Bulk mode should not be enabled when using group processing.
More information available in transformer help.
"""
pass
This will exclude any fme attributes, and create a new attribute called ‘AllEmpty’ (you’ll need to expose this in the PythonCaller) if all found attributes are empty or Null (doesn’t work with Missing)
Python will very likely be faster, but non-Python way is to effectively rename the Attributes into List Attributes. This process will skip over Missing Attributes in building the List. If there are Zero Elements in the output list = All Attributes Missing.
Sample data
Python will very likely be faster, but non-Python way is to effectively rename the Attributes into List Attributes. This process will skip over Missing Attributes in building the List. If there are Zero Elements in the output list = All Attributes Missing.
That is a really clever hack! And quick to implement. Will be using this.