Hello,
I have a CSV file with 70,000 records. It has a header row identified by 10 in the first column. The data rows have 20 in the first column and the footer row has a 99 in the first column. I can split out the three row types, do stuff to the data rows and then combined them all back together again in the correct order – 10, 20, 99.
However, the output file is too big for our thirdparty import routine to handle so I need a way of chunking the data into files with, say, 5,000 records in each. Each file needs the original header row (4 columns), the 5000 data rows (15 columns) and an updated footer row (2 columns) with a 99 and a count of the number of data rows in it. The importer should be able to loop through these smaller files without taxing the underlying JVM.
I have got it mostly working using a fanout on the dataset using @Count(fanout,1,10) which creates 10 folders with a CSV file in each. However the header row gets written to the first file and footer row only gets written to the last file. The other files don’t have the header or footer row. I’ve also tried putting in a ModuloCounter and fanning out on the module_count attribute. Again, the first file gets the header row and another file gets the footer row when each file needs a header and footer row.
Any suggestions from the FME Hive Mind?
Thanks in advance.
