Skip to main content
Solved

How to test every attribute within all records if meet the testers requirments?

  • November 21, 2018
  • 8 replies
  • 366 views
juliarozema
Contributor
Forum|alt.badge.img+7

I am basically doing AttributeValidation but am looking to find values that are blank or empty. Looks like I can't look for blanks or empty's within the AttributeValidation transformer. So, right now I have a testFilter that is checking each field if it is blank or empty. I am using a TestFilter because then I can specify each of the fields that I want to be checked. The problem is that if one record goes through the tester and on the first test we find out that field is blank, then I believe that same record will not go through the rest of the tests. I need to know each field is blank, not just the first one. I am looking for a suggestions on how to do this efficiently.

 

I am basically running a QC process on a large dataset. We have a bunch of business rules that need to be followed. So I am setting FME up to check all those rules. If a rule is broken, then an email will be sent to the person that introduced the error and then they can go back and fix the error.

 

So this one step is looking for fields that were left as blank or spaces so that the data editor can go back and change them to NULLS.

Best answer by ebygomm

If you're only interested in reporting, and you don't need to use this data later, then one possible way is to use a nullattributemapper to set all the nulls to a default value, e.g. null then use the attributevalidator to test for has value, this will then fail the empty attributes and report in a list which have failed

 

View original
Did this help you find an answer to your question?

8 replies

ebygomm
Influencer
Forum|alt.badge.img+34
  • ebygommInfluencer
  • Influencer
  • November 21, 2018

Does the validation rule "has value" not work for you?

juliarozema
Contributor
Forum|alt.badge.img+7
  • juliarozemaContributor
  • Author
  • Contributor
  • November 21, 2018
ebygomm wrote:

Does the validation rule "has value" not work for you?

But I want to know if it doesn't have a value. Is there a way to negate the rule? Oh, I guess I could put only the one rule into the transformer and see what comes out the failed port...

 

I ran it and the failed includes Nulls. I am specifically looking for blanks or ' '
H
Forum|alt.badge.img+2

Hi @juliarozema,

The AttributeFilter transformer may also be of interest to you as it separates Null, missing and empty - but it can still only test on a single attribute at a time.

xiaomengatsafe
Safer
Forum|alt.badge.img+3

Hi @juliarozema I think it might be helpful to share with us what you want to do once you've found the records with empty string or only a space in all the various attributes.

 

For example do you just want to report which attribute has empty or space value, on that record? or do you want to replace them with some other default value? There might be different approaches that are suitable for different subsequent processes you want to perform.
juliarozema
Contributor
Forum|alt.badge.img+7
  • juliarozemaContributor
  • Author
  • Contributor
  • November 21, 2018
xiaomengatsafe wrote:

Hi @juliarozema I think it might be helpful to share with us what you want to do once you've found the records with empty string or only a space in all the various attributes.

 

For example do you just want to report which attribute has empty or space value, on that record? or do you want to replace them with some other default value? There might be different approaches that are suitable for different subsequent processes you want to perform.

Hi Xiaomeng,

 

Good point. I am basically running a QC process on a large dataset. We have a bunch of business rules that need to be followed. So I am setting FME up to check all those rules. If a rule is broken, then an email will be sent to the person that introduced the error and then they can go back and fix the error.

 

So this one step is looking for fields that were left as blank or spaces so that the data editor can go back and change them to NULLS.
takashi
Influencer
  • takashiInfluencer
  • November 21, 2018

If you need to determine if all the target attributes are empty or null or missing, a possible way is to concatenate the values of them and test if the resulting string matches the empty string. For example, this Tester routes the feature to the Passed port if the A, B and C attributes all didn't have any value, routes the feature to Failed port otherwise.

ebygomm
Influencer
Forum|alt.badge.img+34
  • ebygommInfluencer
  • Influencer
  • Best Answer
  • November 22, 2018

If you're only interested in reporting, and you don't need to use this data later, then one possible way is to use a nullattributemapper to set all the nulls to a default value, e.g. null then use the attributevalidator to test for has value, this will then fail the empty attributes and report in a list which have failed

 

petergoedbloed
Contributor
Forum|alt.badge.img+4

Coming late to the party, but would it work to use an attribute exploder, than filter attribute value on your test and then using attribute name to see which records did not pass the test. You can either remove attributes or just keep an ID field to also trace matches back to a certain record.

Reply


Helpful Members This Week

Community Stats

31,263
Posts
118,556
Replies
38,856
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.