Skip to main content

I'm attempting to use FME to inventory folder locations for all shapefiles and file geodatabases, and all feature classes within the file geodatabases to export to an excel file. I am able to identify all shapefiles and geodatabases, and can complete what I'm attempting to do but the performance is really slow.

 

Some background on the workbench I've made - I'm using the feature reader transformer to read the file geodatabases to identify the feature classes (Esri Geodatabase (File Geodb Open API) Reader). The workflow seems to be getting super bogged down at this transformer. I've set the parameters to read only 1 feature, but it still seems to be reading all of the features within the feature class. Is this just the nature of the transformer or is there something I'm missing that's causing the transformer to still read all of the features? I've attached a picture of the feature reader parameters below - any suggestions are greatly appreciated! FeatureReader_parameters

If you only want to fetch the feature class names, why not read the schema features instead of the data features?


This might be more efficient from a PythonCaller using the code below. This assumes each input feature has a 'path_windows' attribute storing the path to the gdb. In the PythonCaller, but sure to use the "Attributes to Expose" parameters to expose the "feature_class" attribute. This should output a feature for each of the feature classes in the gdbs.

import fme
import fmeobjects
 
 
class FeatureProcessor(object):
    """Template Class Interface:
    When using this class, make sure its name is set as the value of the 'Class
    to Process Features' transformer parameter.
    """
 
    def __init__(self):
        """Base constructor for class members."""
        pass
 
    def input(self, feature):
        import arcpy
 
        # Set the workspace for ListFeatureClasses
        arcpy.env.workspace = feature.getAttribute('path_windows')
 
        # Use the ListFeatureClasses function to return a list of feature classes
        featureclasses = arcpy.ListFeatureClasses()
 
        # Create feature for each feature class
        for fc in featureclasses:
            feature.setAttribute("feature_class",fc)
            self.pyoutput(feature)
 
    def close(self):
        """This method is called once all the FME Features have been processed
        from input().
        """
        pass
 
    def process_group(self):
        """When 'Group By' attribute(s) are specified, this method is called 
        once all the FME Features in a current group have been sent to input().
 
        FME Features sent to input() should generally be cached for group-by 
        processing in this method when knowledge of all Features is required. 
        The resulting Feature(s) from the group-by processing should be emitted 
        through self.pyoutput().
 
        FME will continue calling input() a number of times followed
        by process_group() for each 'Group By' attribute, so this 
        implementation should reset any class members for the next group.
        """
        pass
 
    def has_support_for(self, support_type):
        """This method returns whether this PythonCaller supports a certain type.
        The only supported type is fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM.
        
        :param int support_type: The support type being queried.
        :returns: True if the passed in support type is supported.
        :rtype: bool
        """
        if support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM:
            # If this is set to return True, FME will pass features to the input() method that
            # come from a feature table object. This allows for significant performance gains
            # when processing large numbers of features.
            # To enable this, the following conditions must be met:
            #   1) features passed into the input() method cannot be copied or cached for later use
            #   2) features cannot be read or modified after being passed to self.pyoutput()
            #   3) Group Processing must not be enabled
            # Violations will cause undefined behavior.
            return False
 
        return False

 


If you only want to fetch the feature class names, why not read the schema features instead of the data features?

In my exported excel file, I'm also including the geometry type (point, line or polygon) and coordinate system of each geodatabase/feature class and didn't think I could retrieve that information from reading the schema features - am I wrong to think that?


This might be more efficient from a PythonCaller using the code below. This assumes each input feature has a 'path_windows' attribute storing the path to the gdb. In the PythonCaller, but sure to use the "Attributes to Expose" parameters to expose the "feature_class" attribute. This should output a feature for each of the feature classes in the gdbs.

import fme
import fmeobjects
 
 
class FeatureProcessor(object):
    """Template Class Interface:
    When using this class, make sure its name is set as the value of the 'Class
    to Process Features' transformer parameter.
    """
 
    def __init__(self):
        """Base constructor for class members."""
        pass
 
    def input(self, feature):
        import arcpy
 
        # Set the workspace for ListFeatureClasses
        arcpy.env.workspace = feature.getAttribute('path_windows')
 
        # Use the ListFeatureClasses function to return a list of feature classes
        featureclasses = arcpy.ListFeatureClasses()
 
        # Create feature for each feature class
        for fc in featureclasses:
            feature.setAttribute("feature_class",fc)
            self.pyoutput(feature)
 
    def close(self):
        """This method is called once all the FME Features have been processed
        from input().
        """
        pass
 
    def process_group(self):
        """When 'Group By' attribute(s) are specified, this method is called 
        once all the FME Features in a current group have been sent to input().
 
        FME Features sent to input() should generally be cached for group-by 
        processing in this method when knowledge of all Features is required. 
        The resulting Feature(s) from the group-by processing should be emitted 
        through self.pyoutput().
 
        FME will continue calling input() a number of times followed
        by process_group() for each 'Group By' attribute, so this 
        implementation should reset any class members for the next group.
        """
        pass
 
    def has_support_for(self, support_type):
        """This method returns whether this PythonCaller supports a certain type.
        The only supported type is fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM.
        
        :param int support_type: The support type being queried.
        :returns: True if the passed in support type is supported.
        :rtype: bool
        """
        if support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM:
            # If this is set to return True, FME will pass features to the input() method that
            # come from a feature table object. This allows for significant performance gains
            # when processing large numbers of features.
            # To enable this, the following conditions must be met:
            #   1) features passed into the input() method cannot be copied or cached for later use
            #   2) features cannot be read or modified after being passed to self.pyoutput()
            #   3) Group Processing must not be enabled
            # Violations will cause undefined behavior.
            return False
 
        return False

 

Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

You will need to have ArcGIS Pro or ArcMap installed and licensed to run arcpy.


In my exported excel file, I'm also including the geometry type (point, line or polygon) and coordinate system of each geodatabase/feature class and didn't think I could retrieve that information from reading the schema features - am I wrong to think that?

Geometry type yes, but no coordinate system I think. If I had to do this I would use Python like @dustin​ suggested.


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

If you do have an ESRI product installed, make sure your Python Compatibility is set to the python package that came with the ESRI installation.image


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

Okay great, thanks for the help! Last question - is it possible to also use python caller to extract geometry type and coordinate system of each of the feature classes? These are two pieces of information I would like to also export to the excel file which is the main reason I was using the FeatureReader transformer.


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

Sure, if you expand the code @dustin​ expand with the next block:

feature.setAttribute("geometry_type",arcpy.Describe(fc).shapeType)
feature.setAttribute("coordinate_system_name",arcpy.Describe(fc).spatialReference.name)
feature.setAttribute("coordinate_system_code",arcpy.Describe(fc).spatialReference.factoryCode)

Attached working sample.


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

Yes, you can use this code.

# Create feature for each feature class
        for fc in featureclasses:
            coord_sys = arcpy.Describe(fc).spatialReference
            geo = arcpy.Describe(fc).shapeType
            feature.setAttribute("feature_class",fc)
            feature.setAttribute("coord",coor_sys.name)
            feature.setAttribute("geometry",geo)
            
            self.pyoutput(feature)

 


Hi _ddb_, thank you for sharing this. I've attempted to use the scripts you provided and received the following error message:

 

python_caller_errorAny suggestions on resolving it? My apologies, I'm not well versed with this!

Great, thank you both @dustin​ & @nielsgerrits​ for the help. This seems to be working now!


Reply