Question

Dynamic attribute creation and expose?

  • 23 November 2015
  • 8 replies
  • 23 views

Badge
Hello everyone!

 

 

I have one big problem. When I generate attributes dynamically in a workspace, I have to expose them in order for the shapefile writer to actually see and "write" them.

 

 

I have been trying to look up solution to this for quite some time.

 

There is the problem that the incoming source table is generated dynamically, so I have no idea what the attributes are going to be in advance. Manually setting up attribute exposer, or creating list to use in a schema is not possible.

 

 

There are two files in the attachment, one with source csv data, one with attribute transformation I am using and shp folder with shapefiles.

 

 

Desired output would be dynamically generated shapefile, where all attributes from the incoming csv file were present

 

.

 

 

So any ideas or help would be really welcome.

 

Thank you and have a nice day,

 

Radek

8 replies

Userlevel 4
Hi

 

 

The custom transformer SchemaSetter will do most of the work for you, just insert it right after the FeatureMerger. You will probably have to set up a couple of "Prefixes to exclude", e.g. "fme_,multi_,csv_,shape_" to avoid including them in your output shape file.

 

 

If you need more fine grained control, you might also want to look up this article about how to define a schema based on a list attribute.

 

 

David

 

Badge
Hi

 

 

The custom transformer SchemaSetter will do most of the work for you, just insert it right after the FeatureMerger. You will probably have to set up a couple of "Prefixes to exclude", e.g. "fme_,multi_,csv_,shape_" to avoid including them in your output shape file.

 

 

If you need more fine grained control, you might also want to look up this article about how to define a schema based on a list attribute.

 

 

David

 

Thank you so much for this!

 

In this partiucular case I have no control of what attributes will be in the incoming csv file, attributes are generally generated dynamically in the workspace, this transformer (after excluding several prefixes as you have mentioned) did what I needed.

 

 

The article you have linked is known to me, however there are listed only methods where you statically insert which attributes you want to expose / write.
Badge
Hi

 

 

The custom transformer SchemaSetter will do most of the work for you, just insert it right after the FeatureMerger. You will probably have to set up a couple of "Prefixes to exclude", e.g. "fme_,multi_,csv_,shape_" to avoid including them in your output shape file.

 

 

If you need more fine grained control, you might also want to look up this article about how to define a schema based on a list attribute.

 

 

David

 

However it seems there are several attributes missing, I must be doing something wrong in the writer part.

 

 

The problem might be this: The Writer will only check for the list on the first feature of each Feature Type

 

 

Seeing that the list on the first feature is created but is not filled, because the first feature did not have all of the attributes present.

 

 

Each feature in this case can have different attributes. Maybe that is causing problems?
Userlevel 4

However it seems there are several attributes missing, I must be doing something wrong in the writer part.

 

 

The problem might be this: The Writer will only check for the list on the first feature of each Feature Type

 

 

Seeing that the list on the first feature is created but is not filled, because the first feature did not have all of the attributes present.

 

 

Each feature in this case can have different attributes. Maybe that is causing problems?

Yes, not having all the attributes present on the first feature will be a bit more challenging.

Here's a small hack to make the feature with the most attributes arrive at the writer first:

Before the writer, add a PythonCaller with the following code

import fme
import fmeobjects
def FeatureProcessor(feature):
    count = len(feature.getAllAttributeNames())
    feature.setAttribute('_number_of_attributes', count) 

In the PythonCaller, expose the attribute name _number_of_attributes

Then add a Sorter and sort descending (numeric) on _number_of_attributes.

The feature that has the maximum number of attributes should then be the one that defines the output schema.

Note: there are some situations where the above might not suffice, notably where no single feature contains all possible attributes (those cases might need more advanced scripting), so be sure to test before deploying.

David

Badge

Yes, not having all the attributes present on the first feature will be a bit more challenging.

Here's a small hack to make the feature with the most attributes arrive at the writer first:

Before the writer, add a PythonCaller with the following code

import fme
import fmeobjects
def FeatureProcessor(feature):
    count = len(feature.getAllAttributeNames())
    feature.setAttribute('_number_of_attributes', count) 

In the PythonCaller, expose the attribute name _number_of_attributes

Then add a Sorter and sort descending (numeric) on _number_of_attributes.

The feature that has the maximum number of attributes should then be the one that defines the output schema.

Note: there are some situations where the above might not suffice, notably where no single feature contains all possible attributes (those cases might need more advanced scripting), so be sure to test before deploying.

David

Yeah, with almost complete certainity I can already say that none of the features are going to contain all of the attributes. I'll remember this workaround. :)

I have found a way to make the Schemasetter work. I have output the features based on their respective object types, where I can presume that each object in specific object group is going to have the same set of attributes (e.g. roads, buildings,...). When I do this, I can make the dynamic writer take schema from first feature.

 

 

All your answers had been very helpful, thank you very much. :)
Userlevel 2
Badge +17

Yeah, with almost complete certainity I can already say that none of the features are going to contain all of the attributes. I'll remember this workaround. :)

I have found a way to make the Schemasetter work. I have output the features based on their respective object types, where I can presume that each object in specific object group is going to have the same set of attributes (e.g. roads, buildings,...). When I do this, I can make the dynamic writer take schema from first feature.

 

 

All your answers had been very helpful, thank you very much. :)

Hi @drakez,

> I can presume that each object in specific object group is going to have the same set of attributes.

If so, this article may also be helpful

Destination Schema is Derived from a Lookup Table

Takashi

Badge

Yeah, with almost complete certainity I can already say that none of the features are going to contain all of the attributes. I'll remember this workaround. :)

I have found a way to make the Schemasetter work. I have output the features based on their respective object types, where I can presume that each object in specific object group is going to have the same set of attributes (e.g. roads, buildings,...). When I do this, I can make the dynamic writer take schema from first feature.

 

 

All your answers had been very helpful, thank you very much. :)

This link actually is really useful too. The attribute description and actual values are coming from database table, my csv file was just a sample of data I am working with. I cannot share the database connection (obviously), I was just stuck at this part where I had to expose the attributes for shapefile writer.

 

The tables in this database are really specific, I tried deriving destination schema from database tables, but each table has only part of the whole information (there is a table that has information about attributes, however the info about data type of each attribute is stored in another table...), so before coming to the csv I had included here I had to join several tables. I guess that I have to be happy that it works the way @david_r had suggested.

Userlevel 4

Yeah, with almost complete certainity I can already say that none of the features are going to contain all of the attributes. I'll remember this workaround. :)

I have found a way to make the Schemasetter work. I have output the features based on their respective object types, where I can presume that each object in specific object group is going to have the same set of attributes (e.g. roads, buildings,...). When I do this, I can make the dynamic writer take schema from first feature.

 

 

All your answers had been very helpful, thank you very much. :)

For info and/or inspiration: I have done something similar myself, reading the oracle metadata tables (all_tab_columns in particular) to dynamically create the attribute list using Python before sending it to a writer as the output schema definition.

Using the same technique, it is possible to join several tables together in e.g. a SQLCreator and use a PythonCaller to dynamically assemble the attributes{} list based on the query result. You can create the list attribute in one "track" in your workspace and read/process your data in another, then merge them together at the end of your processing, just before hitting the writer.

David

Reply