It would be helpful to have a transformer designed to clean common data problems (selectable from a set) and handle them in standardized ways, such that workspaces would be less likely to fail.
This can be, and surely is frequently, done easily with a chain of existing transformers. But the problem of data with hidden or obvious problems being ingested is pervasive.
In one use case, I have a workspace packaged up as an app on FME Flow. It accepts input from a few fields in the app or as an Excel file. There are more often breaking errors in what users enter than not.
In another, metadata is read in from one source and applied to another, working with XML and esri file geodatabase formats. At different times, characters contained in the metadata have broken the workflow completely.
Suggestions include:
- Trimming leading and trailing spaces
- Removing line breaks
- Converting or removing specified or all unicode characters (e.g. nbsp)
- Escaping or replacing special and/or other problem characters (e.g. `, ‘)
By wrapping a functions to sanitize, replace, and trim data, especially with what are common and often hidden problems, in a single and easy to use transformer, would help to get many workflows off to a better start.
There are probably other characters and faults that I’m not thinking of - a new day, a new data hiccup!
