Skip to main content

Hello everyone,

 

This might be a simple question, but for some reason I can't figure out how to remove rows with only Null Values (Of course I can useTestFiler or Tester, but would be cumbersome work, since I have a lot of columns and I don't want to go into that rabbit hole)

These rows with Null values came into existence, since I read in an excel file with two worksheets with two different schemas. So the column names in worksheet 1, is different compared to the column names in worksheet 2.

Therefore all the values from worksheet 2 have null values. I actually don't need worksheet 2, so I was thinking there might be 2 solutions.

 

1) don't read in worksheet 2 or

2) remove the null values

 

For both of them I don't know the answer.....dramatic sound effects..

So if someone knows the answer, that would be very nice.

If I understand correctly you want to do this dynamic? So input to be defined? One way to do this is to use an AttributeExploder. This will return a feature for each cell with _attr_name and _attr_value. Then you can test for all records where _attr_value has no value. Then merge the results with an Aggregator and merge that back to the original row to shift features with values from features with no values. Be aware that this is memory intensive when having a lot of features and a lot of attributes.


Expanding on @nielsgerrits​ response, in the AttributeExploder you can use the Exploding Type of List which will put your attribute names and values into a consolidate list on the original features. You will need to use the Ignore Attributes Containing parameter to ignore fme format attributes, and any other attributes you aren't testing for (this is a regex input). After the AttributeExploder, you can use a ListConcatentor (making the Separator Character parameter blank). If your test attributes are null, the concatenated attributed from the ListConcatentor will be blank, which can easily be tested for in a Tester or TestFilter.


Expanding on @nielsgerrits​ response, in the AttributeExploder you can use the Exploding Type of List which will put your attribute names and values into a consolidate list on the original features. You will need to use the Ignore Attributes Containing parameter to ignore fme format attributes, and any other attributes you aren't testing for (this is a regex input). After the AttributeExploder, you can use a ListConcatentor (making the Separator Character parameter blank). If your test attributes are null, the concatenated attributed from the ListConcatentor will be blank, which can easily be tested for in a Tester or TestFilter.

Yes, using the list is a far better solution. ListElementCounter instead of ListConcatenator might be quicker an will work as well.


@nielsgerrits​ @dustin​ I feel bad, since I found another solution so only features are read from the first worksheet from excel. I found it based on the following

post.

Using two FeatureReaders and one Sampler in between. I used the followings settings for the sampler and the 2nd FeatureReader with a Constraint based on Features Type to Read: fme_feature_type_name. Now only feature from the first worksheet are read. Since @dustin​ answer is actually more the answer to my question (how to remove Null values), I will select that post as "Select as Best"

afbeeldingafbeelding


Expanding on @nielsgerrits​ response, in the AttributeExploder you can use the Exploding Type of List which will put your attribute names and values into a consolidate list on the original features. You will need to use the Ignore Attributes Containing parameter to ignore fme format attributes, and any other attributes you aren't testing for (this is a regex input). After the AttributeExploder, you can use a ListConcatentor (making the Separator Character parameter blank). If your test attributes are null, the concatenated attributed from the ListConcatentor will be blank, which can easily be tested for in a Tester or TestFilter.

 

Can anyone give more detaild information concerning the configuration of these three transformers, in order to make this workflow work? This seems to be the answer to one of the challanges i tackle at the moment, as well.

I tried to deal with this from a different approch, using a Scriped Value (python) User Parameter as configuration-input (Attributes to Remove) for an “Attribute Remover”, but unfortunatly it did not work out. I’d be very thankful about thoughts on that aswell.

I tried to implement dustins solution for some time now. But i seem to do something wrong. I would be very thankful, if someone could give detailed Information about the configuration of the Attribute Exploder, the List Concatenator and the Tester/Test Filters Parameter-Configuration.

My Goal also is, to remove all Attributes, which contain nothing but Null and empty Values, for my output Data to be left with only relevant attributes.


Hi ​@woelfl

I’ve attached a simple workspace illustrating what I believe was described by dustin & nielsgerrits​. I hope that this serves as a good starting point if you’re looking to dynamically filter out any null values from your dataset. If you don’t need to do this dynamically (ie your schema is the same every time) then you can also simplify things by using just the AttributeValidator instead (I’ve also included this as an option in the attached workspace). 

I hope this helps and please let me know if you have any questions! 

 


Hi ​@joy AttributeValidator Transformer will be able to solve the issue for you.

 


Hi ​@woelfl

I’ve attached a simple workspace illustrating what I believe was described by dustin & nielsgerrits​. I hope that this serves as a good starting point if you’re looking to dynamically filter out any null values from your dataset. If you don’t need to do this dynamically (ie your schema is the same every time) then you can also simplify things by using just the AttributeValidator instead (I’ve also included this as an option in the attached workspace). 

I hope this helps and please let me know if you have any questions! 

 

Hi ​@saraatsafe, thank you very much. this helps a lot!


Reply