Solved

Remove all attributes with empty fields

A
Badge
Hi,

 

 

I have a geodatabase with multiple feature  classes.  The FCs do not all have identical fields.  I want to write a test that removes attributes fields that are empty.  I can't find a way to do this, i.e. select all attribute fields.

 

 

The BulkAttributeRemover seems like the best bet, but not sure how to code the expression.  Also, tried the custom transformer NullAttributeRemover, but the script expects the user to select which attributes to utilise (not a Python user).

 

 

Any suggestions appreciated!  Thanks.
icon

Best answer by takashi 11 November 2015, 04:00

View original

15 replies

takashi
Rank
Userlevel 2
Badge +17
Hi Amit,

 

 

I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count): --- import fmeobjects   # Remove null or empty attributes, # store removed attribute names and the number of them. def removeEmptyAttributes(feature):     removed = []     for name in feature.getAllAttributeNames():         value = feature.getAttribute(name)         if value == None or str(value) == '':             feature.removeAttribute(name)             removed.append(name)     feature.setAttribute('removed_attr_names', ','.join(removed))     feature.setAttribute('removed_attr_count', len(removed)) -----

 

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012   Takashi
takashi
Rank
Userlevel 2
Badge +17

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

A
Badge
Hi Takashi,

 

 

Thanks for the reply and the solution, this worked in Python.  I'm surpised this can't be done natively in FME, but hey-ho problem solved.

 

 

Thanks again.

 

 

Amit
J
Hi Takashi,

 

 

Thanks for the reply and the solution, this worked in Python.  I'm surpised this can't be done natively in FME, but hey-ho problem solved.

 

 

Thanks again.

 

 

Amit

Hi Amit,

Are you able to share Takashi's python file? I attempted to recreate it from the text above but evidently my python skills are not the best.

Thank you,

Jeff

takashi
Rank
Userlevel 2
Badge +17
Hi,

 

 

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

 

 

 

 

Takashi

 

dos_gis
Rank
Badge

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

takashi
Rank
Userlevel 2
Badge +17

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

Hi @dos_gis, there isn't a dedicated transformer that is designed for hiding attribute names, but you can hide unnecessary attributes by the TclCaller (2016) or PythonCaller with setting their "Attributes to Hide" parameter.

mark2atsafe
Rank
Userlevel 4
Badge +25

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

If they are missing then they won't be actual attributes anyway. If you want to hide them on the canvas you can open the Reader feature type and turn them off in the user attributes tab

harmen_kampinga
Badge +2

This was five years ago.... has something changed in the meantime?

 

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

harmen_kampinga
Badge +2

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

to remove attributes containing only <missing> , I've adjusted the script to:

 

import fmeobjects 
 
def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        isNull, isMissing, type = feature.getAttributeNullMissingAndType(name)
        if value == None or value.encode('utf-8') == '' or isMissing == '1' :
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
Still doesn't work....

 

(I'm not very convenient with Python....)

 

 

@takashi : can you help?

 

mark2atsafe
Rank
Userlevel 4
Badge +25

This was five years ago.... has something changed in the meantime?

 

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

As Takashi mentioned, you can use the NullAttributeMapper to convert them from empty to missing (which basically deletes them). But I can see it might be useful to have a specific transformer. Why not post it as an idea on this site? And then other users can upvote if they too would find it useful.

 

takashi
Rank
Userlevel 2
Badge +17

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

Hi @HarmenKampinga, firstly you will have to understand differences among <empty>, <null>, and <missing> in modern FME after the null has been introduced (FME 2014 or later).

 

  • <empty>: a string value which contains no characters.
  • <null>: a special value which indicates the attribute doesn't have any valid value.
  • <missing>: a term (or symbol shown on FME Workbench interface, Log etc.) to indicate that the attribute doesn't exist.
Although some transformers often treat <null> and <missing> as <empty> in fact, they are strictly differentiated internally.

 

In modern FME, this PythonCaller script removes all attributes that store empty (not null).

 

 

def removeEmptyAttributes(feature):
    # The getAllAttributeNames returns names of all existing attributes.
    # It doesn't return names of missing (not existing) attributes.
    for attr in feature.getAllAttributeNames():
        isNull, _, _ = feature.getAttributeNullMissingAndType(attr)
        if not isNull and feature.getAttribute(attr) == '':
            feature.removeAttribute(attr)

 

And, it's equivalent to the NullAttributeValueMapper with this parameters setting. That is, "Map To: Missing" is equivalent to "Remove the attribute".

 

0684Q00000ArMk1QAF.png

 

You will also have to be aware that FME Workbench still keeps displaying names of attributes on its user interface, even if you have created a workspace containing a process to remove those attributes.

 

Whether an attribute exists at run-time won't directly affect whether its name appears on Workbench interface. The attribute names displayed on Workbench interface would be determined statically when you have created the workspace.
P

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

P
Hi,

 

 

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

 

 

 

 

Takashi

 

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

redgeographics
Rank
Userlevel 5
Badge +25

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

It depends on what format you are writing to and how your writer is set up. Some formats, like Shapefile, enforce that all features in a feature type have the same schema, there's simply no way around it. Others, like XML, are much more dynamic.

Reply


Community Stats

28,724
Posts
109,518
Replies
36,564
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.