Question

How to think about custom transformers?

  • 30 November 2022
  • 9 replies
  • 25 views
A

Hey All.

 

I've been looking through the tutorials to set up custom transformers because I wanted a programatic way to define workflows and I was planning on encapsulating/stitching together a Reader, GeometryFilters, and Writer into one custom transformer that took input files, output folder and various other runtime arguments as parameters. The reason for this is to be able to dynamically define the links between workflow steps, whereas now it would require a lot of manual UI work to define a workflow that worked for a file with 1 specific set of features.

 

Is this the right way to be thinking about this problem?

9 replies

redgeographics
Rank
Userlevel 5
Badge +25

In broad terms, yes. I always describe a custom transformer as a collection of transformers that pretends to be a new one. The usage you're describing, reusing a certain static combination of transformers with varying inputs and/or parameters is exactly what custom transformers are designed for.

ebygomm
Rank
Userlevel 1
Badge +21

Personally, I think what you are describing would not be a job for a custom transformer. It sounds more like something where you could develop an entire workspace and then control what parameters it is run with via a workspace runner, fmeserverjobsubmitter or automation or similar.

virtualcitymatt
Rank
Userlevel 4
Badge +26

Yes, depending on your environment you can create linked custom transformers which means you can have a central location for all your customer transformers and just make changes to those.

If using a reader or a writer in a custom transformer it must be a FeatureReader/Writer. And also consider that reading/writing data to disk during a workflow can add quite a bit of processing overhead.

A

In broad terms, yes. I always describe a custom transformer as a collection of transformers that pretends to be a new one. The usage you're describing, reusing a certain static combination of transformers with varying inputs and/or parameters is exactly what custom transformers are designed for.

Do people tend to create these transformers via FME Workbench/Desktop? I've been trying to use the plugin builder SDK and docs but have been struggling a little bit since I haven't completely grasped all the concepts in FME.

​

A

Yes, depending on your environment you can create linked custom transformers which means you can have a central location for all your customer transformers and just make changes to those.

If using a reader or a writer in a custom transformer it must be a FeatureReader/Writer. And also consider that reading/writing data to disk during a workflow can add quite a bit of processing overhead.

So here's an example of the mess I'm trying to avoid / automate.

My thought was I could encapsulate this Reader / Writer / GeometryFilter into one transformer and I can stitch the features in code, given an parameter input of the file names and the output folder name.

 

I'm learning now that it's possible to make custom transformers in the UI as well as using the SDK, but I come from a coding background, so I started with the SDK.

 

Many manual transformer properties

A

Personally, I think what you are describing would not be a job for a custom transformer. It sounds more like something where you could develop an entire workspace and then control what parameters it is run with via a workspace runner, fmeserverjobsubmitter or automation or similar.

Interesting. How would you make it flexible to files with different feature names? Pardon the ignorance, just getting started with this ecosystem. Attached is an example workflow, the feature types are dynamic based on the content of the file, so I'd need a way to reflect those and I only know of doing that in code

 

image

virtualcitymatt
Rank
Userlevel 4
Badge +26

Do people tend to create these transformers via FME Workbench/Desktop? I've been trying to use the plugin builder SDK and docs but have been struggling a little bit since I haven't completely grasped all the concepts in FME.

​

Yes, almost exclusively with FME Workbench.

virtualcitymatt
Rank
Userlevel 4
Badge +26

Interesting. How would you make it flexible to files with different feature names? Pardon the ignorance, just getting started with this ecosystem. Attached is an example workflow, the feature types are dynamic based on the content of the file, so I'd need a way to reflect those and I only know of doing that in code

 

image

Take a look at this tutorial on how to develop dynamic Workflows in FME. FME can handle dynamic schema and table names.

https://community.safe.com/s/article/dynamic-workflow-tutorial-introduction

virtualcitymatt
Rank
Userlevel 4
Badge +26

So here's an example of the mess I'm trying to avoid / automate.

My thought was I could encapsulate this Reader / Writer / GeometryFilter into one transformer and I can stitch the features in code, given an parameter input of the file names and the output folder name.

 

I'm learning now that it's possible to make custom transformers in the UI as well as using the SDK, but I come from a coding background, so I started with the SDK.

 

Many manual transformer properties

You can change the way the input features come into the workspace so they all come in at one feature type. When adding a reader you can check this to make all features come into one.

Here's the tutorial on working with fme in a dynamic way: https://community.safe.com/s/article/dynamic-workflow-tutorial-introduction

image

Reply


Community Stats

28,698
Posts
109,447
Replies
36,548
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.