Skip to main content
  • EN

I have created a workspace which has a prompt for user parameters, as I want the user to be able to run the workflow on different sets of data.

The workspace reads in a set of CSVs which have identical schema, performs some transformations and then writes the data out into a single CSV. The process works for what I want it to do.

 

The problem is when the prompt comes up to run and I choose a different set of CSVs, these CSVs have the same schema as each other but not the same schema as the previous set of data. The attributes that are used in the transformations are the same but other attributes are different and this data does not get read in. I want the reader to read in the new schema each time, but it always retains the schema from the first set of data I used to set up the reader.

 

How can I fix this?

So if I understand your question correctly, for each job/translation you are combining the data of a bunch of CSV files (of one specific schema) together into one large CSV file. But the batch of CSV files for a different job/translation may have have a new (uniform) schema?

 

I guess it's best to use the dynamic reader and writer functionality of FME. You can find more information about dynamic workflows in this article from Safe.

 

There are a couple of approaches you can follow. Personally I prefer using FeatureReaders and FeatureWriters, as they provide a bit more flexibility, and then your workspace could look like this;

imageBut if you choose to use regular Readers and Writers, your workspace could become even more compact by cloning the schema of the Reader to the Writer. Then it could look as simple as this;

imageHope it can help.

 

 

So if I understand your question correctly, for each job/translation you are combining the data of a bunch of CSV files (of one specific schema) together into one large CSV file. But the batch of CSV files for a different job/translation may have have a new (uniform) schema?

 

I guess it's best to use the dynamic reader and writer functionality of FME. You can find more information about dynamic workflows in this article from Safe.

 

There are a couple of approaches you can follow. Personally I prefer using FeatureReaders and FeatureWriters, as they provide a bit more flexibility, and then your workspace could look like this;

imageBut if you choose to use regular Readers and Writers, your workspace could become even more compact by cloning the schema of the Reader to the Writer. Then it could look as simple as this;

imageHope it can help.

 

 

Hi, yes that is exactly what I am looking to do. I tried the dynamic readers approach and it still retains the schema from the first batch of csvs and I don't understand why as this is precisely what this approach is set up to do. Not sure if there is something I'm doing wrong. I've attempted it using both the "generate workspace" approach and by setting the readers and writers as dynamic and neither has worked. my workflow looks like this but when I select a new set of csvs and run it retains the original schema imageNot sure why this is happening

Hi, yes that is exactly what I am looking to do. I tried the dynamic readers approach and it still retains the schema from the first batch of csvs and I don't understand why as this is precisely what this approach is set up to do. Not sure if there is something I'm doing wrong. I've attempted it using both the "generate workspace" approach and by setting the readers and writers as dynamic and neither has worked. my workflow looks like this but when I select a new set of csvs and run it retains the original schema imageNot sure why this is happening

Hi, @ocarey​ ,

You approach seems similar to the WS 'combine_CSV_files_option_2.fmw' that is contained in the zipfile I attached in my previous post. Can you maybe check if that workspace would work for you (I hope you can open the recent fmw, I'm using FME 2021.2.2.0 )?

 

I think your issue could be related to the naming of the feature types and schema features. When Adding the CSV Reader, what setting did you use for the Feature Type Name(s)? The default 'From Format Name', or did you switch to 'From File Name(s)'?

imageSimilarly, what kind of settings did you use for the writer feature types?

imageDid you maybe change the CSV File name parameter to a new (user parameter) value?

 

For example, if you use the default Reader FT name option 'From Format Name', all CSV feature types will be named 'CSV'. But then if you change the CSV File name at the writer, and leave the Schema Sources and Schema Definition Name to the default values, you will get a mismatch. In this case your writer would be expecting a schema with your newly set/configured CSV file name (Default for Schema Definition Name is 'Default from CSV File name above'), but your reader is still using the feature type name 'CSV'.

However, in this case you wouldn't get any features to be written, where you mention that the features are being written but with the original schema.

 

Nevertheless I think these Reader/Writer FT settings could be the cause of undesired translation behavior. Maybe you can add some screenshots to show how (the feature types of) your Dynamic Reader and Writer are configured? Or even better, maybe you can add the workspace to this case so I can have a look at what might be going wrong?

Reply


Community Stats

30,353
Posts
115,359
Replies
38,089
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.