Skip to main content
Solved

Remove all attributes with empty fields

A
Forum|alt.badge.img
Hi,

 

 

I have a geodatabase with multiple feature  classes.  The FCs do not all have identical fields.  I want to write a test that removes attributes fields that are empty.  I can't find a way to do this, i.e. select all attribute fields.

 

 

The BulkAttributeRemover seems like the best bet, but not sure how to code the expression.  Also, tried the custom transformer NullAttributeRemover, but the script expects the user to select which attributes to utilise (not a Python user).

 

 

Any suggestions appreciated!  Thanks.

Best answer by takashi

Hi,

 

 

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

 

 

 

 

Takashi

 

View original
Did this help you find an answer to your question?

15 replies

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 3, 2013
Hi Amit,

 

 

I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count): --- import fmeobjects   # Remove null or empty attributes, # store removed attribute names and the number of them. def removeEmptyAttributes(feature):     removed = []     for name in feature.getAllAttributeNames():         value = feature.getAttribute(name)         if value == None or str(value) == '':             feature.removeAttribute(name)             removed.append(name)     feature.setAttribute('removed_attr_names', ','.join(removed))     feature.setAttribute('removed_attr_count', len(removed)) -----

 

If you aren't familar with Python, start from: Python and FME Basics New Python FME Objects API in FME 2012   Takashi
takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 3, 2013

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

A
Forum|alt.badge.img
  • amit
  • Author
  • August 5, 2013
Hi Takashi,

 

 

Thanks for the reply and the solution, this worked in Python.  I'm surpised this can't be done natively in FME, but hey-ho problem solved.

 

 

Thanks again.

 

 

Amit
J
  • jrags01
  • November 10, 2015
amit wrote:
Hi Takashi,

 

 

Thanks for the reply and the solution, this worked in Python.  I'm surpised this can't be done natively in FME, but hey-ho problem solved.

 

 

Thanks again.

 

 

Amit

Hi Amit,

Are you able to share Takashi's python file? I attempted to recreate it from the text above but evidently my python skills are not the best.

Thank you,

Jeff

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • Best Answer
  • November 11, 2015
Hi,

 

 

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

 

 

 

 

Takashi

 

dos_gis
Contributor
Forum|alt.badge.img+3
  • dos_gisContributor
  • Contributor
  • March 14, 2016

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • March 15, 2016
dos_gis wrote:

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

Hi @dos_gis, there isn't a dedicated transformer that is designed for hiding attribute names, but you can hide unnecessary attributes by the TclCaller (2016) or PythonCaller with setting their "Attributes to Hide" parameter.

mark2atsafe
Safer
Forum|alt.badge.img+43
  • mark2atsafeSafer
  • Safer
  • March 15, 2016
dos_gis wrote:

Hi, I'd like to take this a step farther and remove the attribute columns that only contain "Missing" attributes. Any ideas?

If they are missing then they won't be actual attributes anyway. If you want to hide them on the canvas you can open the Reader feature type and turn them off in the user attributes tab

harmen_kampinga
Forum|alt.badge.img+2

This was five years ago.... has something changed in the meantime?

 

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

harmen_kampinga
Forum|alt.badge.img+2
takashi wrote:

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

to remove attributes containing only <missing> , I've adjusted the script to:

 

import fmeobjects 
 
def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        isNull, isMissing, type = feature.getAttributeNullMissingAndType(name)
        if value == None or value.encode('utf-8') == '' or isMissing == '1' :
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
Still doesn't work....

 

(I'm not very convenient with Python....)

 

 

@takashi : can you help?

 

mark2atsafe
Safer
Forum|alt.badge.img+43
  • mark2atsafeSafer
  • Safer
  • August 3, 2018
harmen_kampinga wrote:

This was five years ago.... has something changed in the meantime?

 

I like to have a transformer which can check for attributes with no data in it, empty fields. So you can decide to remove them.

All about data quality and minimalization of attributes needed.

As Takashi mentioned, you can use the NullAttributeMapper to convert them from empty to missing (which basically deletes them). But I can see it might be useful to have a specific transformer. Why not post it as an idea on this site? And then other users can upvote if they too would find it useful.

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • August 4, 2018
takashi wrote:

Hi Amit,

 

 

  I think a PythonCaller is the quickest way.

 

This script, for example, removes null or empty attributes, then stores removed attribute names and the number of them as new attributes (removed_attr_names, removed_attr_count):
# 2016-03-15: Restored indentations, added a Note
# Remove null or empty attributes,
# store removed attribute names and the number of them.

import fmeobjects 

def removeEmptyAttributes(feature):
    removed = []
    for name in feature.getAllAttributeNames():
        value = feature.getAttribute(name)
        if value == None or str(value) == '': # See the following Note.
            feature.removeAttribute(name)
            removed.append(name)
    feature.setAttribute('removed_attr_names', ','.join(removed))
    feature.setAttribute('removed_attr_count', len(removed))
        
# Note: FMEFeature.getAttribute() vs. <null>
# If the specified attribute stores <null>, the method returns
# - None in FME 2014 and earlier,
# - the empty string in FME 2014 SP1 or later.
# This script example was posted in 2013.
# In 2014 SP1+, the method returns None only if the attribute is missing. 
# You can use FMEFeature.getAttributeNullMissingAndType() method
# to differentiate <null>, <missing>, and <empty>, in FME 2014+.

If you aren't familar with Python, start from:    Python and FME Basics New Python FME Objects API in FME 2012         Takashi

Hi @HarmenKampinga, firstly you will have to understand differences among <empty>, <null>, and <missing> in modern FME after the null has been introduced (FME 2014 or later).

 

  • <empty>: a string value which contains no characters.
  • <null>: a special value which indicates the attribute doesn't have any valid value.
  • <missing>: a term (or symbol shown on FME Workbench interface, Log etc.) to indicate that the attribute doesn't exist.
Although some transformers often treat <null> and <missing> as <empty> in fact, they are strictly differentiated internally.

 

In modern FME, this PythonCaller script removes all attributes that store empty (not null).

 

 

def removeEmptyAttributes(feature):
    # The getAllAttributeNames returns names of all existing attributes.
    # It doesn't return names of missing (not existing) attributes.
    for attr in feature.getAllAttributeNames():
        isNull, _, _ = feature.getAttributeNullMissingAndType(attr)
        if not isNull and feature.getAttribute(attr) == '':
            feature.removeAttribute(attr)

 

And, it's equivalent to the NullAttributeValueMapper with this parameters setting. That is, "Map To: Missing" is equivalent to "Remove the attribute".

 

0684Q00000ArMk1QAF.png

 

You will also have to be aware that FME Workbench still keeps displaying names of attributes on its user interface, even if you have created a workspace containing a process to remove those attributes.

 

Whether an attribute exists at run-time won't directly affect whether its name appears on Workbench interface. The attribute names displayed on Workbench interface would be determined statically when you have created the workspace.
P

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

P
takashi wrote:
Hi,

 

 

This subject was posted in 2013. You can use the NullAttributeMapper in 2014+.

 

 

 

 

Takashi

 

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

redgeographics
Celebrity
Forum|alt.badge.img+47
pascalfilippi wrote:

Hi @takashi i try to use this transformer but he didnt delete my different Attribute, i don t know if after i need to put my Writter in a special way like Dynamic or what ever, to have less attribute than in the reader. Do you know how to use this transformer ?

It depends on what format you are writing to and how your writer is set up. Some formats, like Shapefile, enforce that all features in a feature type have the same schema, there's simply no way around it. Others, like XML, are much more dynamic.

Reply


Helpful Members This Week

Community Stats

30,908
Posts
117,105
Replies
38,708
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.