Question

Remove attributes in dynamic reader

  • 26 September 2017
  • 18 replies
  • 34 views
D
Badge

Hi,

I am translating a DGN file to File GDB using dynamic schema.

In DGN reader, i can see so many extra attributes like attribute.color, attribute.length etc. I want to ignore those attributes in file GDB writer.

I am using attribute keeper but it is not working in case of Dynamic Schema,

Please guide.

Thanks in advance.

18 replies

K
Badge +1

Hi @dheeraj

Have you tried adding the attributes that you want to remove explicitely in the writer?

david_r
Rank
Userlevel 4

Attached is a sample custom transformer that can be used to remove schema attributes: removeschemaattributes.fmwt

This solution is slightly more flexible than removing the attribute names on the writer, although the effect is the same.

Also, please consider voting for this idea to make the AttributeRemover work with dynamic schemas:

https://knowledge.safe.com/idea/48070/attributeremover-etc-to-work-with-schema-features.html

D
Badge

Hi @dheeraj

Have you tried adding the attributes that you want to remove explicitely in the writer?

Hi i am unable to see those attributes in above dropdown list which i want to remove in File GDB but i can see those extra attributes in DGN reader and also getting those attributes in FileGDB.

 

Also want to inform that i am not getting any extra attributes if i am using Static schema and attribute keeper.

 

 

david_r
Rank
Userlevel 4
Hi i am unable to see those attributes in above dropdown list which i want to remove in File GDB but i can see those extra attributes in DGN reader and also getting those attributes in FileGDB.

 

Also want to inform that i am not getting any extra attributes if i am using Static schema and attribute keeper.

 

 

Sounds like you should try the custom transformer I posted above. It does not rely on attributes being exposed.
K
Badge +1

Hi @dheeraj

Have you tried adding the attributes that you want to remove explicitely in the writer?

Hi @dheeraj,

 

 

Which version of FME are you using? The screenshot above was taken in FME 2016.1. If you use an older version of FME, the 'attributes to remove' option might be hidden in the 'Schema Definition' option (screenshot below is from FME 2015.0).

D
Badge

I am using FME 2017 and i am able to see the option but not able to see those attributes in the list.

takashi
Rank
Userlevel 2
Badge +17

I am using FME 2017 and i am able to see the option but not able to see those attributes in the list.

You have to manually enter attribute names to be removed into the "Attributes to Remove" list.

 

Just be aware that the GDB writer will replace every "." in the original attribute names with "_" automatically, so you need to enter "attribute_color" to remove "attribute.color", for example.
takashi
Rank
Userlevel 2
Badge +17

Attached is a sample custom transformer that can be used to remove schema attributes: removeschemaattributes.fmwt

This solution is slightly more flexible than removing the attribute names on the writer, although the effect is the same.

Also, please consider voting for this idea to make the AttributeRemover work with dynamic schemas:

https://knowledge.safe.com/idea/48070/attributeremover-etc-to-work-with-schema-features.html

Hi @david_r, thanks for sharing the custom transformer. Regarding the Python script embedded in the transformer, I'm afraid that the number of list elements would not be reduced automatically only by resetting them with the setAttribute method. Probably the original list elements (at least excess part) have to be removed explicitly before resetting them. Could you please check this?

 

david_r
Rank
Userlevel 4
Hi @david_r, thanks for sharing the custom transformer. Regarding the Python script embedded in the transformer, I'm afraid that the number of list elements would not be reduced automatically only by resetting them with the setAttribute method. Probably the original list elements (at least excess part) have to be removed explicitly before resetting them. Could you please check this?

 

Good catch, thanks for letting me know about this behaviour! I've fixed the attached template.
D
Badge

I am using FME 2017 and i am able to see the option but not able to see those attributes in the list.

Hi @takashi, I have tried manual entry as well but did not get desired result.

 

L
Badge

Hi @dheeraj

have you tried BulkAttributeRemover? You could e.g. remove all attributes that start with igds_ and all attributes that start with fme_ with this transformer.

oliver.morris
Rank
Badge +10

Attached is a sample custom transformer that can be used to remove schema attributes: removeschemaattributes.fmwt

This solution is slightly more flexible than removing the attribute names on the writer, although the effect is the same.

Also, please consider voting for this idea to make the AttributeRemover work with dynamic schemas:

https://knowledge.safe.com/idea/48070/attributeremover-etc-to-work-with-schema-features.html

@david_r

 

Thank
 you for the custom transformer, this is incredibly helpful. I amended 
it slightly to basically look for contains matches but I should probably
 make this better so as not to remove attributes by accident.

 

 

import fmeobjects

class RemoveSchemaAttrs(object):
    def input(self,feature):
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            for single_remove in attrs_to_remove:
                if single_remove in attr_name:
                    fmeobjects.FMELogFile().logMessageString(
                        'Removing schema attribute "%s" from schema "%s"' % 
                        (attr_name, fme_feature_type_name), fmeobjects.FME_WARN)
                    del attr_names[attr_index]
                    del attr_fme_datatypes[attr_index]
                    del attr_native_datatypes[attr_index]
        feature.removeAttribute('attribute{}.name')
        feature.removeAttribute('attribute{}.fme_data_type')
        feature.removeAttribute('attribute{}.native_data_type')
        feature.setAttribute('attribute{}.name', attr_names)
        feature.setAttribute('attribute{}.fme_data_type', attr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', attr_native_datatypes)
        
        self.pyoutput(feature) 

 

S
@david_r

 

Thank
 you for the custom transformer, this is incredibly helpful. I amended 
it slightly to basically look for contains matches but I should probably
 make this better so as not to remove attributes by accident.

 

 

import fmeobjects

class RemoveSchemaAttrs(object):
    def input(self,feature):
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            for single_remove in attrs_to_remove:
                if single_remove in attr_name:
                    fmeobjects.FMELogFile().logMessageString(
                        'Removing schema attribute "%s" from schema "%s"' % 
                        (attr_name, fme_feature_type_name), fmeobjects.FME_WARN)
                    del attr_names[attr_index]
                    del attr_fme_datatypes[attr_index]
                    del attr_native_datatypes[attr_index]
        feature.removeAttribute('attribute{}.name')
        feature.removeAttribute('attribute{}.fme_data_type')
        feature.removeAttribute('attribute{}.native_data_type')
        feature.setAttribute('attribute{}.name', attr_names)
        feature.setAttribute('attribute{}.fme_data_type', attr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', attr_native_datatypes)
        
        self.pyoutput(feature) 

 

I could use the Workbench very well. But I have noticed that an attribute that follows another attribute, i.e. has an index 1 higher, is not removed.

Could you please help

marnickcle
Rank
Badge +3

Attached is a sample custom transformer that can be used to remove schema attributes: removeschemaattributes.fmwt

This solution is slightly more flexible than removing the attribute names on the writer, although the effect is the same.

Also, please consider voting for this idea to make the AttributeRemover work with dynamic schemas:

https://knowledge.safe.com/idea/48070/attributeremover-etc-to-work-with-schema-features.html

hi

@david_r​ ​  , I encountered this problem today that I could not remove an attribute in the schema. I wanted to try your custom transformer, but I can't seem to locate it. Is this still available on the Knowledge Base? Thanks in advance!

david_r
Rank
Userlevel 4

hi

@david_r​ ​  , I encountered this problem today that I could not remove an attribute in the schema. I wanted to try your custom transformer, but I can't seem to locate it. Is this still available on the Knowledge Base? Thanks in advance!

The attachment seems to have been lost in "the great forum migration" (one of them...), but fortunately I still had a local copy:

https://www.dropbox.com/s/qbil2mb1i2ahn0r/removeschemaattributes.fmwt?dl=1

Just be aware that the Python script is a bit crude, it could certainly be improved upon.

oliver.morris
Rank
Badge +10

hi

@david_r​ ​  , I encountered this problem today that I could not remove an attribute in the schema. I wanted to try your custom transformer, but I can't seem to locate it. Is this still available on the Knowledge Base? Thanks in advance!

clearly your weblink half life it a lot longer than mine!

marnickcle
Rank
Badge +3

hi

@david_r​ ​  , I encountered this problem today that I could not remove an attribute in the schema. I wanted to try your custom transformer, but I can't seem to locate it. Is this still available on the Knowledge Base? Thanks in advance!

Thanks a bunch!

J
@david_r

 

Thank
 you for the custom transformer, this is incredibly helpful. I amended 
it slightly to basically look for contains matches but I should probably
 make this better so as not to remove attributes by accident.

 

 

import fmeobjects

class RemoveSchemaAttrs(object):
    def input(self,feature):
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            for single_remove in attrs_to_remove:
                if single_remove in attr_name:
                    fmeobjects.FMELogFile().logMessageString(
                        'Removing schema attribute "%s" from schema "%s"' % 
                        (attr_name, fme_feature_type_name), fmeobjects.FME_WARN)
                    del attr_names[attr_index]
                    del attr_fme_datatypes[attr_index]
                    del attr_native_datatypes[attr_index]
        feature.removeAttribute('attribute{}.name')
        feature.removeAttribute('attribute{}.fme_data_type')
        feature.removeAttribute('attribute{}.native_data_type')
        feature.setAttribute('attribute{}.name', attr_names)
        feature.setAttribute('attribute{}.fme_data_type', attr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', attr_native_datatypes)
        
        self.pyoutput(feature) 

 

I see the same thing. It looks like this is because the enumerated list is being modified in the for loop. For example, if you're in the loop at index '2' and the item at index '2' is removed, the item that *was* at index '3' moves down to index '2', and '4' moves down to '3', etc. In the next iteration of the loop, you've now moved on to index '3', but what used to be there has moved.

 

One way to accommodate this is to reverse the order of the list and work your way down instead of up. That is, change this:

for attr_index, attr_name in enumerate(attr_names):

to this:

 

for attr_index, attr_name in reversed(list(enumerate(attr_names))):

 

Reply


Community Stats

28,692
Posts
109,433
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.