Should this be a Batch Process or...?

Hey Loren,

Great question, thank you for reaching out! There are a few possibilities that you could consider. Transformers can often help when processing multiple and large files - if you haven’t looked into the different types of transformers I’ve compiled a small list and included some articles that could benefit you.

The Feature Holder transformer may be of assistance - The FeatureHolder can temporarily hold features until all have been processed and released at once. This can be useful in ensuring that subsequent transformers receive all features simultaneously.

Many Transformers also use GroupBy Parameters. This can help ensure they are processed as groups rather than individually.

Also using a DataInspector could be used to verify all features are being read correctly from your source files before they are processed.
 

The following articles might also help you find the answer that suits you best!

How Blocking (or Feature-Holding) Transformers Alter the Flow of Features in FME. An article showing how blocking transformers alter the flow of Features.

Feature Caching and Performance. An article on Feature Caching and how it affects performance and your workspace.

FME Batch Processing Methods. An article explaining the different types of Batch Processing.

Let me know if this helps and if you need anything else I’m happy to assist!

Donal.

Hi Donal,

Thanks for the info.  This has been an issue both going out and coming in.  I had never used the FeatureHolder before, but using one for each feature type does NOT work.  I’ll see if funnelling everything into one and then breaking them out after right before the featurewriter works.

To me, this should be bullt-in to the feature writer.  Why wouldn’t I want all my data to end up in the place I designated, right?  What I am trying to do does not seem like a ‘Batch’ operation, it’s a single Excel file output.  We are not fanning out or calling distinct, separate operations.  It’s just a bunch of data processed into one file.  I’ll post whatever solution I come up with.  Wish me luck!

Best,

-Loren

Here is what I have found with using FeatureHolder:

It seemed to work, all the features made it, BUT each sheet in the Excel workbook has a different schema, which caused major issues.  Every sheet had all the attributes, so it messed up my columns.   

Despite my best efforts, I could not acheive a ‘Dynamic’ workflow.  One of the sheets did not show up at all.  Using this tutorial , I tried to differentiate schemas, but it only worked as far as putting the data on the correct sheet. 

Here are some screenshots of where the data goes through the featureholder and onto the writer.  Notice that I have a disabled writer with all the sheets (on the right).  Doing it that way would entail manually editing which attributes go to their respective sheet.  Not very dynamic.    Am I missing something?