There is currently no simple way to have truly dynamic attribute mapping.
I would like the ability to provide an input file, a mapping file (old attribute, new attribute) and get an output that is identical to the input except for the attribute names.
While the schemaMapper allows you to dynamically rename attributes referencing an external lookup table, that lookup table is hardcoded, you cannot use a parameter to specify it.
Additionally you cannot dynamically write out the changed attributes. All the schemaMapper examples assume hard coded attributes on the writer, which can lead to issues if the mapping table is updated and the writer is not. Both automatic and dynamic mode use the original schema unless schema from schema feature is selected.
While one can use the SchemaSetter custom transformer to generate the new schema feature, it has the flaw that all strings are mapped to varchar(256) , leading to significant bloat in the output file when the original attribute might only be a few characters.
Two updates to the schema mapper would allow this dynamic ability without having to script it in python.
- Allow the Schema mapping table to be set by a user parameter, or even an attribute.
- Have an output port on the SchemaMapper with an updated schema feature
