Remove all attributes with empty fields

Hi,

I have a geodatabase with multiple feature classes. The FCs do not all have identical fields. I want to write a test that removes attributes fields that are empty. I can't find a way to do this, i.e. select all attribute fields.

The BulkAttributeRemover seems like the best bet, but not sure how to code the expression. Also, tried the custom transformer NullAttributeRemover, but the script expects the user to select which attributes to utilise (not a Python user).

Any suggestions appreciated! Thanks.

Page 1 / 1

Hi Amit,

I think a PythonCaller is the quickest way.

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count): --- import fmeobjects # Remove null or empty attributes, # store removed attribute names and the number of them. def removeEmptyAttributes(feature): removed = v] for name in feature.getAllAttributeNames(): value = feature.getAttribute(name) if value == None or str(value) == '': feature.removeAttribute(name) removed.append(name) feature.setAttribute('removed_attr_names', ','.join(removed)) feature.setAttribute('removed_attr_count', len(removed)) -----

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012 Takashi

Hi Amit,

I think a PythonCaller is the quickest way.

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):

# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = /]
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012 Takashi

Hi Takashi,

Thanks for the reply and the solution, this worked in Python. I'm surpised this can't be done natively in FME, but hey-ho problem solved.

Thanks again.

Amit

Hi Takashi,

Thanks for the reply and the solution, this worked in Python. I'm surpised this can't be done natively in FME, but hey-ho problem solved.

Thanks again.

Amit

Hi Amit,

Are you able to share Takashi's python file? I attempted to recreate it from the text above but evidently my python skills are not the best.

Thank you,

Jeff

Hi,

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

Takashi

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

Hi @dos_gis, there isn't a dedicated transformer that is designed for hiding attribute names, but you can hide unnecessary attributes by the TclCaller (2016) or PythonCaller with setting their "Attributes to Hide" parameter.

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

If they are missing then they won't be actual attributes anyway. If you want to hide them on the canvas you can open the Reader feature type and turn them off in the user attributes tab

This was five years ago.... has something changed in the meantime?

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

Hi Amit,

I think a PythonCaller is the quickest way.

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):

# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = /]
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012 Takashi

to remove attributes containing only <missing> , I've adjusted the script to:

import fmeobjects 
 
def removeEmptyAttributes(feature):
    removed = �]
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        isNull, isMissing, type = feature.getAttributeNullMissingAndType(name)
        if value == None or value.encode('utf-8') == '' or isMissing == '1' :
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))

Still doesn't work....

(I'm not very convenient with Python....)

@takashi : can you help?

This was five years ago.... has something changed in the meantime?

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

As Takashi mentioned, you can use the NullAttributeMapper to convert them from empty to missing (which basically deletes them). But I can see it might be useful to have a specific transformer. Why not post it as an idea on this site? And then other users can upvote if they too would find it useful.

Hi Amit,

I think a PythonCaller is the quickest way.

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):

# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = /]
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012 Takashi

Hi @HarmenKampinga, firstly you will have to understand differences among <empty>, <null>, and <missing> in modern FME after the null has been introduced (FME 2014 or later).

<empty>: a string value which contains no characters.
<null>: a special value which indicates the attribute doesn't have any valid value.
<missing>: a term (or symbol shown on FME Workbench interface, Log etc.) to indicate that the attribute doesn't exist.

Although some transformers often treat <null> and <missing> as <empty> in fact, they are strictly differentiated internally.

In modern FME, this PythonCaller script removes all attributes that store empty (not null).

def removeEmptyAttributes(feature):
    # The getAllAttributeNames returns names of all existing attributes.
    # It doesn't return names of missing (not existing) attributes.
    for attr in feature.getAllAttributeNames():
        isNull, _, _ = feature.getAttributeNullMissingAndType(attr)
        if not isNull and feature.getAttribute(attr) == '':
            feature.removeAttribute(attr)

And, it's equivalent to the NullAttributeValueMapper with this parameters setting. That is, "Map To: Missing" is equivalent to "Remove the attribute".

You will also have to be aware that FME Workbench still keeps displaying names of attributes on its user interface, even if you have created a workspace containing a process to remove those attributes.

Whether an attribute exists at run-time won't directly affect whether its name appears on Workbench interface. The attribute names displayed on Workbench interface would be determined statically when you have created the workspace.

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

Hi,

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

Takashi

It depends on what format you are writing to and how your writer is set up. Some formats, like Shapefile, enforce that all features in a feature type have the same schema, there's simply no way around it. Others, like XML, are much more dynamic.

Reply

Community Stats

Reply

Community Stats

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded