I haven't used the SchemaMapper in a long while, mostly because it's not always as flexible as I'd like and I find it easier and faster to get the exact behavior I want using Python.

A pattern I've used successfully a few times have been a PythonCaller with a class method. Typically, I'll read the contents of a CSV or similar rules file in the __init__() method and assign it to an instance dict object. In the input() method I'll query the feature for attributes and apply rules using the instance dict.

This gets me very good performance and a lot of flexibility in e.g. using regex-based rules etc that aren't supported by the SchemaMapper.

I have used the Joiner transformer quite a bit. Join the information from the Excel/CSV file to the features and work from there to get the required output.

Probably not as fast as the Python solution, but it does not require Python experience (and that is not my strongest point).

Probably not as good with complex schema mapping either.

I do like using the DatabaseJoiner for schema mappings -- that was the way years ago we used to do it. And the prefetch was created for that type of scenario. (Well, at for reading MGE data but I digress.) SchemaMapper is fancier and more powerful for sure, but at the expense of more complexity. We could take a deep breath and revisit the SchemaMapper interface and underlying code one more time but I'm not convinced it is time well spent -- unless we get enhancement suggestions that are quite simple to understand (like introducing more operators than implied equality -- < > regexp etc). Perhaps an Idea could be filed and then we can collect thoughts and votes over there?

Hi @nic_ran​ , @erik_jan​  and @david_r​ !

We're revisiting the SchemaMapper over here, to see how we might be able to shave down it's long tooth. In digging into this transformer, it's evident there are a lot of questions about what exactly a new-n-improved SchemaMapper looks like. We're beginning to look at supporting simple functions and conditionals but still need to investigate/verify the following:

• Which functions and conditionals would be used? What would be helpful?
• Would these statements be specified in the external lookup table, the configuration fields, or apply to the incoming feature values themselves?
• What kind of UI help would the author expect for writing these statements?

Any and all feedback welcomed on these above points, if you would be game to share! I would greatly appreciate it!

Cheers,

Annabelle

Hi Annabelle, it's been many years since I last used the SchemaMapper, but my primary reason for stopping to use it was that all the settings were hard-coded into the workspace, there was no easy way to use it in dynamic workspaces where some settings or conditionals would depend on the input. Also, the wizard interface was clunky to work with.

Finally I seem to remember that it was fairly slow for large-ish data volumes, but I could be wrong, so please take it with a grain of salt.

My typical use case would be for translating attribute names before output using dynamic schema. For example, when reading from a database you get the technical column names back, but often the users want something more "humanly readable" in their output file, so if they could maintain e.g. an Excel sheet with translations that would result in renaming both the schema feature and the actual data features, that would be amazing. As it is, I have to resort to Python to accomplish this.

Regarding the GUI, my pet peeve is when you just want to adjust a small setting in an existing transformer, but it then requires access to the full dataset before you can accept the changes (e.g. the FeatureReader). If that can be avoided, it would be very helpful.

I hope this helps.

Hi Annabelle,

like David, it's been a number of years since I've used the SchemaMapper and the last time I had a need for it I actually wrote my own version because the standard SchemaMapper couldn't do what I needed it to. My general opinion of the current version is that it's too complicated for the average user to bother with but not powerful enough for advanced users.

The main issues I had with it at the time I last used it I think I've covered in my original post asking for feedback, i.e. you can't do conditional mapping with multiple operators (I'd like to see capabilities similar to the Tester transformer), and you can't parameterise the input. Both of those shortcomings relate to the functionality of the mapping rules and limitations on what can actually be achieved in schema mapping with the transformer. There is also no way in the SchemaMapper to map features based on a spatial predicate test (e.g. spatial intersect, area, length, etc.).

I fully understand that a schema mapper with the above level of functionality significantly blurs the line between functionality that could be encapsulated in a rules database and what can be done directly in an FME workspace - and where do you draw the line - but it could certainly pave the way toward some very generic workspaces.

You do raise another good point, though. The interface is not easy to use or understand. I very much like the idea of an external database to hold the schema mapping information, as this can help keep the mapping logic editable without requiring FME knowledge (e.g. by a business analyst). I think there's definitely room for improvement in the existing interface in this regard.

It might also be worth considering splitting the SchemaMapper into 2 separate transformers:

1. external database rules (as in the existing SchemaMapper)
2. fully inline transformer that allows rules to be built within the workspace (ideally, making use of cached data values)

That's all I've got off the top of my head right now but I may add more detail later, once I've had a chance to review my old work on this.

Nic