Solved

Bulk Mode in PythonCaller

  • 24 January 2023
  • 3 replies
  • 51 views

Userlevel 6
Badge +36

I see an idea to add/support bulk mode in the PythonCaller has just been updated to "Released" - https://community.safe.com/s/bridea/a0r4Q00000HbriAQAR/support-bulk-mode-in-pythoncaller

 

I'm wondering what the usage of the bulk mode is in the PythonCaller, by design the PythonCaller iterates over each feature individually, therefore going against the idea of bulk mode.

 

 

 

 

icon

Best answer by markatsafe 24 January 2023, 21:03

View original

3 replies

Userlevel 1
Badge +2

@hkingsbury​ Great question. There is no real benefit within the python itself. The main benefit will be that if you have feature tables entering the PythonCaller they will be preserved on exit. So other transformers can still benefit from the feature table.

In the PythonCaller template there is a function "has_support_for". If you set this to return 'True' then feature tables will be preserved.

So good to use if you have large numbers of feature being processed in your PythonCaller.

Userlevel 6
Badge +36

@hkingsbury​ Great question. There is no real benefit within the python itself. The main benefit will be that if you have feature tables entering the PythonCaller they will be preserved on exit. So other transformers can still benefit from the feature table.

In the PythonCaller template there is a function "has_support_for". If you set this to return 'True' then feature tables will be preserved.

So good to use if you have large numbers of feature being processed in your PythonCaller.

Thanks Mark :)

Userlevel 2
Badge +12

There is no real benefit within the python itself. The main benefit will be that if you have feature tables entering the PythonCaller they will be preserved on exit. So other transformers can still benefit from the feature table.

In the PythonCaller template there is a function "has_support_for". If you set this to return 'True' then feature tables will be preserved.

So good to use if you have large numbers of feature being processed in your PythonCaller.

For future reference, I’ll note that “preserve” is the important part, here. Setting the has_support_for flag for FME_SUPPORT_FEATURE_TABLE_SHIM has the effect of splitting a feature table into individual features that are tagged as belonging to a bulk mode feature table, and will be reassembled at output.

The C/C++ SDK describes it like this (from fmeobjects\cpp\fmesupporttype.h):

enum SupportType
{
// ...
// Whether a plugin supports feature tables via a "shim", which passes the feature table to the
// plugin as if it was features (i.e. instead of getting one feature with an IFMEFeatureTable
// geometry, the plugin will get N features). This can allow plugins to preserve feature tables
// without explicitly handling them, which may improve performance. Note that some calls on the
// IFMEFeature interface are not supported for shim features, and calling them will cause the
// feature table to be split into individual features, potentially losing the performance
// benefit.
// Factories that opt in for shim support are required to either output or destroy all input
// features during the process() call (i.e. the factory cannot store a feature and use it later).
FME_SUPPORT_FEATURE_TABLE_SHIM = 1,

// ...

};

The fmeobjects Python library doesn’t let you create bulk mode features like this, which would make this feature only useful for attribute manipulation on existing features (something that the PythonFactory doc calls “Single Input-Single Output style”). However, there is a workaround abusing the “preserve” mechanism that lets you pass Python-created features to a factory pipeline that would output them in bulk mode, and then returning the output of that pipeline as the output of your PythonCaller. Since these features are now tagged as being part of a Bulk Mode feature table, they get reassembled by FME before it sends them downstream, something that can tremendously cut down on time spent waiting for the next transformer to accept/finish processing the features your PythonCaller outputs.

Reply