Summary:
I need assistance applying a header (a feature that only contains field names) to a large dataset that is formatted without field names.
Data:
I have a massive dataset (~17 million records) that has been split into 17 chunks, each one containing ~1 million records. In each of these files there are ~150 unique fields, none of which have a proper name. So when loading a chunk of 1 million records, the field names default to col0, col1, col2 etc. Then I have a separate header file, which has the correct ordering of field names that needs to be applied over the default values.
Problem:
Applying the header to a large number of attributes stored in different datasets, without typing them individually.
Bad solutions:
- Attribute manager / attribute renamer would work, however its tedious and not a good use of time to type out the field names of all 150 attributes.
- Bulk Attribute Renamer has a cool import button that can be used. Unfortunately the import option only works on one half (input or output) and you would still have to manually type the default 'colX' for each field. Additionally, importing from a feature can only be sorted alphanumerically, and loading from the header (already in the correct order) auto-sorts attributes and loses any significant organization.
- I could use R or Stata, but because the subsequent workflow is in FME, I'd rather do it all here.
Great solutions:
Use the workspace runner to loop through a path that contains the chunked files.
Any thoughts or ideas are appreciated! This seems like a fairly trivial or mundane task that I've not yet figured out how to do in FME. Thank you!
