Hello,
Can anyone explain the difference shown below? Inside a standard workflow, the SchemaScanner captures the order of the attribute names correctly as they appear on the input file. However, when used inside of a custom transformer, the incoming attribute names are sorted alphanumerically. Does anyone have any ideas why this is happening?
As Spock would say, "Fascinating".
Sadly, I know exactly why this is happening. And I also cannot see any easy or helpful way around it. Let me explain and then perhaps some idea can emerge.
The reason this ultimately is going on is that fundamentally, at translation run time, data flowing through FME does not have any concept of column order. The actual data rows that fly through and are processed do not themselves have any ordering of attributes. The SchemaScanner itself works on those attributes that are on data features, and has no way of knowing what the original source ordering of those attributes was. So in the absence of that, it sorts them on output.
And that was the best we could do, and how the first iterations of the SchemaScanner were delivered a year or so ago.
But customers like you didn't like this and wanted the original order preserved. So the team dug in deep and made use of the fact that at design time, while you're working in Workbench, Workbench knows what the original order of the columns was:
So when you put down a SchemaScanner on the "main canvas", we quietly send that ordering information into the SchemaScanner as part of its configuration. If you right click and copy a SchemaScanner transformer on a workbench canvas, and then paste into your favorite text editor, you'll see a line like this:
TEMPLATE_SCHEMA { IdNr,uint32,GPSTime,uint64,Latitude,real64,Longitude,real64 }
Which we generated from the information on the canvas and passed on, via this hint line, to the FME translation engine so it can use it at translation run time to give a hint as to the preferred order of the attributes.
What happens if you put the SchemaScanner inside a custom transformer? Well, in a custom transformer we don't know what attributes are going to be coming in, and so inside a custom transformer we aren't able to show them, nor do we know what they are. (This is also true because a single custom transformer might be used in multiple places in the translation, where the schemas coming in could be quite different).
Bottom line -- this means that a SchemaScanner inside a custom transformer won't have any TEMPLATE_SCHEMA hint, and so can only go for sorting the attributes when it kicks out a schema.
I have a couple of ideas for working around this *for particular scenarios*, so if you're able to share the key parts of your workspace, or more details on the scenario you're solving, I'm game to explore the solution space. The easiest of course is just to do the SchemaScanning outside of the custom transformer, if that is a possibility (and I apologize for that in advance...)
Sorry, and hope this explanation helps.
Hi Dale!
Thanks for getting back to my request with such a thorough response. I thought it might have something to do with the "run time" issue.
I'm creating a custom transformer that I hope to present at this year’s FME User Conference. I've called it 'KMLBalloonStyler' and it will allow the user to create an optimized KML output with mutiple colour themes / style options for the Attribute table. It will also provide additional features within the KML ballon such as navigation arrows, date, feature count etc. A key part of the workflow is getting the Attribute names in the correct order, as this will allow me to generate a custom HTML table in the correct order as well.
One potential solution might be to have the user specify the order through a user parameter. As you can see below the user can select all the attribute names and uncheck 'Sorted' which will change the attributes from being sorted alphabetically back into their correct order. It's a bit cumbersome but it's the only workaround I can think of.
The issue is the user can also select the Attribute values, which I don't need for the workflow and would need to filter out, which presents another set of challenges. Ideally the user would only be able to select the attribute names. Perhaps this issue can be solved using a PythonCaller, I know Mark seemed to think this was possible - https://community.safe.com/s/question/0D54Q00008SQCbeSAH/how-to-only-select-attribute-names-for-attributelist-parameter.
Let me know if you have any better ideas. Happy to share the custom transformer with you if that helps.