Remove attributes based on external list / Use attribute value as regex in BulkAttributeRemover

Hi ​@lilywong,

I tested several methods, including BulkAttributeRemover, BulkAttributeRenamer, and PythonCaller, to remove the attributes, but I reached the same conclusion as you - the attributes that contain PRIVATE are still present, and their values appear to be <missing>.

I’ll file a ticket with our development team and keep you updated on the progress.

Thank you for bringing this to our attention!

Emma

Hi,

I just ran into the exact same problem yesterday. So i am very happy to see, that this has already being brought to the attention of the creme de la creme.

In my case I am trying to build a workflow for dynamically removment of all these attributes in my files, which contain nothing but <null> or <missing> values. So i placed a Python Caller (building a regex expression after checking the data for strictly empty/null containing attributes) followed by an AttributeExposer in front of the BulkAttributeRemover. Even the <null> values get turned to <missing> values when handing over the regex as an attributevalue, the attributes itsself however remain in the table.

Cheers, Tobi

Hi ​@lilywong ,

I think BulkAttributeRemover definitely works as expected, even if the regex is given by an attribute value.

In the Table View of FME, <missing> means that the attribute does not exist, the attribute name is just being shown in FME Workbench interface. Since FME Workbench cannot know the attribute value (regex) when you edit the workspace, the target attribute names won't disappear.

I would recommend you to determine if an attribute exits, see the Feature Infromation window rather than the Canvas interface or the Table View.

takashi wrote:

Hi ​@lilywong ,

I think BulkAttributeRemover definitely works as expected, even if the regex is given by an attribute value.

In the Table View of FME, <missing> means that the attribute does not exist, the attribute name is just being shown in FME Workbench interface. Since FME Workbench cannot know the attribute value (regex) when you edit the workspace, the target attribute names won't disappear.

I would recommend you to determine if an attribute exits, see the Feature Infromation window rather than the Canvas interface or the Table View.

 

Hi @takashi,

thank you very much for your suggestions. I followed your advice and took a closer look at the Feature Information window to better understand which attributes remain after using the BulkAttributeRemover and which do not. Indeed, in both cases—whether I pass the regex manually as a string or as an attribute value—the attributes I intended to remove do not appear in the Feature Information window.

However, it seems that the BulkAttributeRemover (BAR) still handles the two methods of passing the regex differently.

When writing the BulkAttributeRemover’s output to a FeatureClass in an Esri Geodatabase (the same format as the Reader), the attributes to be removed are not written to the output table if the regex is entered manually. Additionally, they do not appear in the Canvas interface or the Table View.

On the other hand, if the regex is passed dynamically as an attribute value, the attributes not only show up in the Canvas interface and Table View, but the values of the attributes to remove are successfully deleted as expected.

Interestingly, the attributes to remove are still written to the FeatureClass table, even if they don't appear in the Feature Information window after the BulkAttributeRemover process. In this case, the BAR seems to not fully fulfill its intended purpose, correct? It seems to me, that it is doing only half the job in this case.

Thank you very much again,

Tobias

If you have configured User Attributes in the destination writer feature type with Automatic mode, FME would statically configure destination schema according to attributes exposed on the Workbench interface when you are editing the workspace, regardless of the attributes actually would exist or not at run-time.

In your case, I think you should consider confiure destination schema with Dynamic mode rather than Automatic mode. I would recommend you to start from Tutorial: Dynamic Workflows to learn about Dyanmic Workflow.

Hi ​@takashi,

thank you very much. I already configured the destination schema of my writer with dynamic mode. But that doesn’t seem to do the trick either.

I’ll work through the tutorial.

Thanks

Hi ​@woelfl ,

If you are attempting to configure the destination schema with schema feature read by FeatureReader, consider using ListElementFilter from FME Hub to modify the schema definition (i.e. "attribute{}" list) with the same regex used in the BulkAttributeRemover. 

Hi ​@lilywong, 

thanks to ​@takashi pointing into the direction of the Dynamic Workflow - Tutorial, I found a solution to this Problem. See the attached workbench-template for my solution:

Since this is my first template to share ever, I hope I did everything right and its ready to use. If any problems occur, please let me know.

The Template is meant as an example for a simple dynamic workflow for BulkAttributeRemoving.

Input is an XLSX-Sheet containing columns(attributes) with pulic data, private data, Null-data and a regular expression (regex).

The regex was written in order to indentify the attriubute containing the regex itself, aswell as all the attributes containing private data.

The input-table is connected to the BulkAttributeRemover (BAR).

The regex is handed in form of an attribute value.

The BARs Output is connected to an SchemaScanner with default configuration. (Note that Missing/Null/Empty Attributes will be ignored).

The SchemaScanners Output and Schema are connected to an Microsoft Excel-Writers Input.

While Adding the Writer its Sheet-Definition has been Set to „Dynamic“.

Further Configurations have been made:

- Setting the Schema Sources to: „Schema From Schema Feature“

- Setting the Schema Definition Name to : „fme_feature_type_name“

I hope this helps you as well as it helps me. Since my concern was only to remove Missing/Null Attributes the SchemaScanner works very well for me. But I’am confident it is able to deliver a solution to your difficulties aswell.

Happy workbenching,

Tobias