Skip to main content

I think what I want to do should be simple enough, but I'm struggling to figure out the syntax in the PythonCaller. 

 

I'd appreciate any help with the following:

 

I'm writing to an Enterprise Geodatabase using the FeatureWriter, and want to then grant two roles appropriate privileges to these feature classes.

 

This is simple enough in a standalone ArcPy script with something like the below:

fcList = arcpy.ListFeatureClasses()
for fc in fcList:
    arcpy.ChangePrivileges_management(fc, "Viewer", "GRANT", "AS_IS")
    arcpy.ChangePrivileges_management(fc, "Editor", "GRANT", "GRANT")

However, I can't figure out how to pass each feature class name to the PythonCaller (in the place of 'fc', above) .

 

As I mentioned, I think this should be pretty simple, but I'm obviously missing something...

Hi Lindsay, you need to access the parameter value dictionary like here:

https://community.esri.com/community/open-platform-standards-and-interoperability/blog/2019/03/21/using-arcgis-pros-python-environment-in-etl-tools

 


On the FeatureWriter "Summary" port, connect a ListExploder and configure it to iterate over the list _feature_type{}.

Then connect a PythonCaller with something like:

import fmeobjects 
import arcpy

arcpy.env.workspace = 'change_me.sde'
 
def change_permissions(feature):
    fc = feature.getAttribute('name')
    arcpy.ChangePrivileges_management(fc, "Viewer", "GRANT", "AS_IS")
    arcpy.ChangePrivileges_management(fc, "Editor", "GRANT", "GRANT")

This will execute the two arcpy.ChangePrivileges_management() functions once for each feature class output by the FeatureWriter.


Thanks so much for the responses!

I ended up going with David's approach, using the ListExploder.

The privileges are now being applied via the Python code, but it takes over 5 minutes to process each feature class.

The features are being written to an Amazon RDS from a desktop PC, so the whole process is inherently slower than with a local database, but 5 minutes to apply privileges to each feature class seems a little excessive...

I had read that the PythonCaller will process commands for every feature passed to it if you use a function and that you should use a Python class if processing per feature class.

Should I wrangle the above code into the class template?

My proof of concept workbench is set to only read 2 feature classes and, looking at the workbench, it certainly seems that only one record is being passed to the PythonCaller per feature class:

 

Alternatively, could I (should I) iterate through each feature class in a shutdown Python script?

 

Thanks again for all the help!


I'm suspecting that the bottleneck is with arcpy, not FME. In fact, the PythonCaller hardly does anything at all but to pass on parameters to arcpy. You can verify this by commenting out the two last lines of the PythonCaller.

Try disabling the PythonCaller and manually executing the arcpy.ChangePrivileges_management calls from the ArcGIS Python console after FME has terminated and see how long it takes there.

Rewriting the PythonCaller to use a class definition rather than a function will have no impact on the performance in itself.


Hi David,

Thanks for the suggestion.

I ran a Python script to apply the privileges in isolation, and then ran the same process in ModelBuilder.

Interestingly (and disappointingly), the results are the opposite of what I had expected:

  • ModelBuilder was fastest, applying privileges to both feature classes in 1 minute 12 seconds.
  • The standalone Python script took 6 minutes 32 seconds.
  • The FME workbench, running nearly identical code, took 9 minutes 28 seconds.

Given that ModelBuilder is essentially running the same Python code as the other two solutions, I can't figure out why there is such a large discrepancy.

I had hoped to use FME to automate a data upload process, currently being run through ModelBuilder, but given these results, I'm second guessing myself.

I think the overall process will still be faster in FME, but this is the first time I've ever found FME to be slower than an Esri product when performing a similar task, and it has left me scratching my head...


Hi David,

Thanks for the suggestion.

I ran a Python script to apply the privileges in isolation, and then ran the same process in ModelBuilder.

Interestingly (and disappointingly), the results are the opposite of what I had expected:

  • ModelBuilder was fastest, applying privileges to both feature classes in 1 minute 12 seconds.
  • The standalone Python script took 6 minutes 32 seconds.
  • The FME workbench, running nearly identical code, took 9 minutes 28 seconds.

Given that ModelBuilder is essentially running the same Python code as the other two solutions, I can't figure out why there is such a large discrepancy.

I had hoped to use FME to automate a data upload process, currently being run through ModelBuilder, but given these results, I'm second guessing myself.

I think the overall process will still be faster in FME, but this is the first time I've ever found FME to be slower than an Esri product when performing a similar task, and it has left me scratching my head...

Logically, I find it very difficult to understand those findings (assuming that you're doing all the tests on the same geodatabase, with the same feature class containing the same data volume)., in particular the large discrepancy between the ModelBuilder and the standalone Python script. You may want to ask on the Esri forums about that, which could perhaps give some clues for why FME is even slower.


Logically, I find it very difficult to understand those findings (assuming that you're doing all the tests on the same geodatabase, with the same feature class containing the same data volume)., in particular the large discrepancy between the ModelBuilder and the standalone Python script. You may want to ask on the Esri forums about that, which could perhaps give some clues for why FME is even slower.

Hi David,

Yes, the data source and destination are the same in all three cases, so I can't understand my findings either.

This and some other behaviour makes me think there is something strange going on with our connection to the RDS, so I'm looking now at running the workbench on an AWS desktop, which should have much better throughput to the RDS.

Thanks once again for your assistance with the PythonCaller, I'll post an update here for completion once I find a solution to the performance issue.


Reply