Skip to main content

Hi everyone,

I have a workflow in which I have to change the schema dynamically. I was able to write the result I want into a CSV file by using a FeatureWriter (or a CSV writer) and setting Dynamic Schema Definition with Schema sources as "Schema From Schema Feature". The features passed in have list attributes name and fme_data_type.

The table in the output CSV file is correctly formatted into the new schema. However, the data from the output port of FeatureWriter is in the same schema format as the input to it, (not in schema format written in CSV). I want to change schema in FME so that I can process it further. Is there some setting in FeatureWriter that Im missing? or any Alternative workflow to change the schema ?

 

Thanks in advance,

Ashish

Hi Ashish,

I'm not sure of any setting that would help but I do have a suggestion. What you could do is add a FeatureReader after the Writer to read in the data with the correct Schema. It's an extra step but should solve the issue.

 

Brian

Hi @brianatsafe,

Thanks for your suggestion.

I tried adding a FeatureReader after the Writer. However, I could not set the FeatureReader properly so that it dynamically reads the schema. It seems the attributes for each feature is read in, but not exposed (However, I can't use AttributeExposer as I would not know the attribute names beforehand).

Also, the filenames of the csv file that I'm writing (and reading again) are based on an attribute value. (If I explicitly select the csv file, then FeatureReader correctly produces the schema.)

 

The output from the FeatureReader is as follows:

 

My workspace is as follows:


Hi @brianatsafe,

Thanks for your suggestion.

I tried adding a FeatureReader after the Writer. However, I could not set the FeatureReader properly so that it dynamically reads the schema. It seems the attributes for each feature is read in, but not exposed (However, I can't use AttributeExposer as I would not know the attribute names beforehand).

Also, the filenames of the csv file that I'm writing (and reading again) are based on an attribute value. (If I explicitly select the csv file, then FeatureReader correctly produces the schema.)

 

The output from the FeatureReader is as follows:

 

My workspace is as follows:

Thanks @ashish_man. Excellent context here. I understand the issue a bit deeper and I'd like to know more about what you would like to do.

Question: downstream from the FeatureWriter, what specifically do you want to achieve with FME? Let us know what you are trying to do and I'll provide any suggestions. (for example, are you trying to edit Attribute Names if a specific name is in the file...or anther type or process)

 

 

You hit the nail on the head that FME doesn't understand the schema because it is being read in at runtime rather than design so it makes sense that workbench doesn't provide you with exposed attributes.

 

 


Thanks @ashish_man. Excellent context here. I understand the issue a bit deeper and I'd like to know more about what you would like to do.

Question: downstream from the FeatureWriter, what specifically do you want to achieve with FME? Let us know what you are trying to do and I'll provide any suggestions. (for example, are you trying to edit Attribute Names if a specific name is in the file...or anther type or process)

 

 

You hit the nail on the head that FME doesn't understand the schema because it is being read in at runtime rather than design so it makes sense that workbench doesn't provide you with exposed attributes.

 

 

Thanks for the support @brianatsafe. My objective is to automate reading tables from a number of Autocad drawings from different locations, and present it in a way useful to other user groups. The final output format could be a database containing info from all locations. Mostly the tables in the Autocad drawings are just a bunch of lines and text put together that appear as a table. Moreover, the title, attribute names, content, location or the number of the tables on the dwg are not consistent. The only thing common are few key words of the attribute names of the table.

There may be many different way to solve this problem, but I started by searching these keywords, their neighbour lines and after a bunch of text location based transformations, I was able to reproduce the tables in a CSV file by setting dynamic schema definition. The CSV file is just an temporary file that had to be created for changing the schema of the table dynamically. My next step is to generalise the tables (renaming attributes to standard format, merging tables etc) so that it can be put into a database. But for it, I have to dynamically read the tables.

I might need to avoid dynamic reading of CSV, by doing all necessary transformations before writing to the CSV. Please suggest.

Thanks,

Ashish

 


Thanks @ashish_man. Excellent context here. I understand the issue a bit deeper and I'd like to know more about what you would like to do.

Question: downstream from the FeatureWriter, what specifically do you want to achieve with FME? Let us know what you are trying to do and I'll provide any suggestions. (for example, are you trying to edit Attribute Names if a specific name is in the file...or anther type or process)

 

 

You hit the nail on the head that FME doesn't understand the schema because it is being read in at runtime rather than design so it makes sense that workbench doesn't provide you with exposed attributes.

 

 

Thank you for that context. Super helpful and this sounds like an interesting project! Creating those tables from drawings is a huge accomplishment in my books. Anyways, I think part of the problem is that you need some user interaction between reading the original data and then once the table is understood you want to further process it. Unless you have a number of specific rules that you can specify in the workspace, it may be easier to have a user drive the process.

 

One idea comes to mind (and we have an example of it here: https://knowledge.safe.com/articles/1116/easygeocoder-data-driven-self-serve-data-transform.html), is to read the data and then have the user decide what to do with it. FME Server does make this easier b/c you can run the first process and stream the results to a second webpage to ask the user questions. Check out that demo and see if that would help.

 

 

Anyone else have other ideas?


Reply