Skip to main content
Solved

Generic workspaces with custom processing configured at runtime

nic_ran
Contributor
Forum|alt.badge.img+16

This may be more of an idea for an FME enhancement but I'm curious to know if anyone thinks this is possible, or if there's a way to do it...

I want to create a truly generic/dynamic workflow. The source and target data and schema requirements can be driven by database tables, as can reprojection requirements. The majority of the time this will likely be sufficient to achieve what we need, however, there may be cases in which a simple source->target mapping is insufficient and we need some custom ETL transformations. Is there any way that the name of a custom transformer can be passed in as a parameter at runtime and have that transformer called during processing?

This functionality could be partially met by using the WorkspaceRunner or the FMEServerJobSubmitter but both of those transformers fire up a whole new translation for each feature that enters them. What I really need is a "TransformerRunner" transformer. Something that can dynamically include a (custom) transformer into a workspace at runtime...

Best answer by nic_ran

For what it's worth, I've implemented a successful test case for this that works on both FME Desktop and FME Server, with some minor limitations.

The attached zip file contains a very simple workspace and some custom transformers for testing the solution.

dynamic-transformer-caller.zip

General Solution

I created a Python scripted parameter which copies the required transformer over from a custom resources folder to the FME transformers folder at run time. I initially tried this with a startup Python script but had to switch to a scripted parameter. There are obviously some subtle differences in runtime between the startup scripts and the scripted parameters. The startup script runs too late in the workspace initialisation for the copied custom transformer to get used.

Limitations

The solution works well but has some limitations that I'm aware of:

  • each custom transformer instance must have the exact same name defined for the "Name" value in the Transformer Properties, although the actual file names can (and should) be different.
  • each custom transformer must also expose the same input parameters. If different parameters are required, then they can be passed in as feature attributes and exposed using the AttributeExposer inside the custom transformer.

Strengths

One of my main concerns was how FME Server would behave in the event of a conflict. Due to the way the solution is implemented, the custom transformer has to be copied over to a shared resources directory at run time. This directory is shared by all users of FME Server and there is no way to prevent other users from running the same workspace at the same time, using a different custom transformer. My brief testing indicates that the file copy is very fast and occurs at the time the workspace is first loaded. Immediately after the file copy has occurred the transformer is loaded into memory for the workspace to reference. Copying a different transformer into the shared directory after the copy has occurred does not change the internal reference of the transformer being used by the workspace and each workspace uses the correct version of transformer.

View original
Did this help you find an answer to your question?

10 replies

redgeographics
Celebrity
Forum|alt.badge.img+47

Yes, sort of, I think.

If you have the custom transformers laid out in your workspace in parallel streams you can route features to them by testing on a user parameter.

But you have to have the processing designed and in your workspace for this to work (either directly in the workspace or in custom transformers). If you want something truly dynamic you could attempt to generate mapping files on the fly, store them in a temp location and call them using a WorkspaceRunner (I'm not even sure about how this would work on FME Server, I think you have to register the mapping files with a repository for them to become available)

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • March 11, 2017

I think @redgeographics idea of routing via a published parameter test is the best one if you have a set of well known ahead of time transformations you want to apply.

Though I'd be surprised if it would be truly dynamic -- that is, you want the user to supply a custom transformer and you'll apply it -- you could accomplish such a thing by copying the custom transformer on top of whatever one the workspace was authored to use. I also played around and the FMEFunctionCaller allows you to inject an arbitrary line into the resulting workspace mapping file so a super duper advanced old timer (say, a SpatialDirect veteran) could use that to INCLUDE a mapping file fragment and then apply hand-coded transformation dynamically.

Not sure I'd recommend it though.

david_r
Evangelist
  • david_rEvangelist
  • March 13, 2017

If you're not intimidated by some Python programming, your can accomplish a lot using the fmeobjects API inside a PythonCaller. It's really easy to analyse features based on attributes, geometry types, etc and to act accordingly. Starting with FME 2017 even raster features are supported.

The fmeobjects documentation gives a pretty good overview of the capabilities:

http://docs.safe.com/fme/html/FME_Objects_Python_API/index.html

redgeographics
Celebrity
Forum|alt.badge.img+47
fmelizard wrote:

I think @redgeographics idea of routing via a published parameter test is the best one if you have a set of well known ahead of time transformations you want to apply.

Though I'd be surprised if it would be truly dynamic -- that is, you want the user to supply a custom transformer and you'll apply it -- you could accomplish such a thing by copying the custom transformer on top of whatever one the workspace was authored to use. I also played around and the FMEFunctionCaller allows you to inject an arbitrary line into the resulting workspace mapping file so a super duper advanced old timer (say, a SpatialDirect veteran) could use that to INCLUDE a mapping file fragment and then apply hand-coded transformation dynamically.

Not sure I'd recommend it though.

Peter Laulund could probably do it :)

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • March 20, 2017
fmelizard wrote:

I think @redgeographics idea of routing via a published parameter test is the best one if you have a set of well known ahead of time transformations you want to apply.

Though I'd be surprised if it would be truly dynamic -- that is, you want the user to supply a custom transformer and you'll apply it -- you could accomplish such a thing by copying the custom transformer on top of whatever one the workspace was authored to use. I also played around and the FMEFunctionCaller allows you to inject an arbitrary line into the resulting workspace mapping file so a super duper advanced old timer (say, a SpatialDirect veteran) could use that to INCLUDE a mapping file fragment and then apply hand-coded transformation dynamically.

Not sure I'd recommend it though.

Thanks Dale, I like this idea. I don't want to be restricted to pre-defined transformers in a workspace because this would ideally be driven by information coming from database tables and everything would happen at runtime.

 

 

I'll play around with the FMEFunctionCaller to see if I can get it to do what I need but I think the idea of copying the required transformer over a pre-defined transformer would probably be the simplest solution.

 

 

It would be really nice if Safe could implement a "CustomTransformerCaller" transformer. :)

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • March 20, 2017
fmelizard wrote:

I think @redgeographics idea of routing via a published parameter test is the best one if you have a set of well known ahead of time transformations you want to apply.

Though I'd be surprised if it would be truly dynamic -- that is, you want the user to supply a custom transformer and you'll apply it -- you could accomplish such a thing by copying the custom transformer on top of whatever one the workspace was authored to use. I also played around and the FMEFunctionCaller allows you to inject an arbitrary line into the resulting workspace mapping file so a super duper advanced old timer (say, a SpatialDirect veteran) could use that to INCLUDE a mapping file fragment and then apply hand-coded transformation dynamically.

Not sure I'd recommend it though.

On second thoughts, not sure the "copying over" approach will be the best solution as I won't be able to guarantee the workspace is run in isolation (ie. multiple requests may come in at the same time, each requiring a different custom transformer). Perhaps the FMEFunctionCaller is the best approach after all...

 

 

Time to start playing!

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • March 31, 2017
fmelizard wrote:

I think @redgeographics idea of routing via a published parameter test is the best one if you have a set of well known ahead of time transformations you want to apply.

Though I'd be surprised if it would be truly dynamic -- that is, you want the user to supply a custom transformer and you'll apply it -- you could accomplish such a thing by copying the custom transformer on top of whatever one the workspace was authored to use. I also played around and the FMEFunctionCaller allows you to inject an arbitrary line into the resulting workspace mapping file so a super duper advanced old timer (say, a SpatialDirect veteran) could use that to INCLUDE a mapping file fragment and then apply hand-coded transformation dynamically.

Not sure I'd recommend it though.

For what it's worth, I've implemented a successful test case for this on FME Desktop.

 

 

I created a Python scripted parameter which copies the required transformer over at run time. I initially tried this with a startup Python script but there are obviously some subtle differences in runtime between the startup scripts and the scripted parameters. The startup script runs too late in the workspace initialisation for the copied custom transformer to get used.

 

 

Next test... implement this on FME Server.

 

 

I'll post my results once done.

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • March 31, 2017
david_r wrote:

If you're not intimidated by some Python programming, your can accomplish a lot using the fmeobjects API inside a PythonCaller. It's really easy to analyse features based on attributes, geometry types, etc and to act accordingly. Starting with FME 2017 even raster features are supported.

The fmeobjects documentation gives a pretty good overview of the capabilities:

http://docs.safe.com/fme/html/FME_Objects_Python_API/index.html

David, thanks for this suggestion, however, I want a solution that will be easy to implement and maintain in the future. I'm personally not averse to using custom Python but it's not very transparent what's going on and I want other developers to be able use whatever solution gets implemented in a "plug and play" manner. Being able to have them reference a custom FME transformer seems the simplest solution.

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • March 31, 2017
redgeographics wrote:

Yes, sort of, I think.

If you have the custom transformers laid out in your workspace in parallel streams you can route features to them by testing on a user parameter.

But you have to have the processing designed and in your workspace for this to work (either directly in the workspace or in custom transformers). If you want something truly dynamic you could attempt to generate mapping files on the fly, store them in a temp location and call them using a WorkspaceRunner (I'm not even sure about how this would work on FME Server, I think you have to register the mapping files with a repository for them to become available)

Thanks redgeo. Parallel streams won't work because I want the solution to be generic and not hard-coded into the workspace. Using a workspace runner won't work because I want the solution to run against every feature in the controlling workspace. A workspace runner would fire up a new FME for each feature and would be very slow.

 

 

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • Best Answer
  • April 3, 2017

For what it's worth, I've implemented a successful test case for this that works on both FME Desktop and FME Server, with some minor limitations.

The attached zip file contains a very simple workspace and some custom transformers for testing the solution.

dynamic-transformer-caller.zip

General Solution

I created a Python scripted parameter which copies the required transformer over from a custom resources folder to the FME transformers folder at run time. I initially tried this with a startup Python script but had to switch to a scripted parameter. There are obviously some subtle differences in runtime between the startup scripts and the scripted parameters. The startup script runs too late in the workspace initialisation for the copied custom transformer to get used.

Limitations

The solution works well but has some limitations that I'm aware of:

  • each custom transformer instance must have the exact same name defined for the "Name" value in the Transformer Properties, although the actual file names can (and should) be different.
  • each custom transformer must also expose the same input parameters. If different parameters are required, then they can be passed in as feature attributes and exposed using the AttributeExposer inside the custom transformer.

Strengths

One of my main concerns was how FME Server would behave in the event of a conflict. Due to the way the solution is implemented, the custom transformer has to be copied over to a shared resources directory at run time. This directory is shared by all users of FME Server and there is no way to prevent other users from running the same workspace at the same time, using a different custom transformer. My brief testing indicates that the file copy is very fast and occurs at the time the workspace is first loaded. Immediately after the file copy has occurred the transformer is loaded into memory for the workspace to reference. Copying a different transformer into the shared directory after the copy has occurred does not change the internal reference of the transformer being used by the workspace and each workspace uses the correct version of transformer.

Reply


Most helpful members this week

Community Stats

30,846
Posts
116,940
Replies
38,670
Members
Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.