With libraries like DuckDB in your Python environment, as is the case where you either have ArcGIS Pro or Enterprise on your FME machine, or you’re using ArcGIS Data Interoperability, then it is easy to retrieve potentially millions of features from an S3-API compliant object store or other web source in seconds, but it’s a bottleneck to send them on into the workspace one by one.
I would like to see performance like reading CSV files brought to PythonCreator.
There is a possibly related existing idea Introduce a Python Dataframe Creator/Transformer but I don’t want to confound using dataframes with this idea, which is fundamentally about performance.
Dataframes might be how this idea is implemented, but my guess is that would be a heavy lift for Safe, another way might be a way to output an aggregate feature.
