Solved

AttributeManager - Removed Lists reappearing in downstream WorkBench Transformer partial results?

B
Badge +3

FME 2018.1.1.2

 

Is this expected behaviour? I've set 2 Lists to be removed from the WorkBench features by AttributeManager. Opening the Partial Results of the Output Port in Data Inspector, they are indeed removed.

Yet, in the very next Transformer (happens to be TopologyBuilder) the 2 Lists reappear again on the Features when the Output Port is inspected in Data Inspector, complete with the original list values? Not only that, but trying to remove them with a subsequent AttributeManager, AttributeKeeper etc. does not work, because these Transformers don't think that either of the Lists exist anymore in the Schema and hence won't appear in the selection of Attributes/Lists available to modify! These Lists continue down the Workflow in the Partial Results but remain invisible in the Schema.

 

Further, if I replace the AttributeManager with an AttributeKeeper, the Lists stay removed (and I guess is a workaround?). I'm guessing this may be a bug in how AttributeManager tries to implement a bulk feature mode in only remove Lists from the notional Schema rather than from the actual WorkBench feature data? (or at least in the Partial results) Whereas AttributeKeeper forces a split in the Schema to create a new branch of data?

icon

Best answer by mark2atsafe 10 January 2020, 17:08

View original

16 replies

fmelizard
Rank
Userlevel 4
Badge +13

Hi @bwn How did you remove the list with the AttributeManager? Were the list attributes actually removed from the features or did they just appear to be removed from under the AttributeManager in your workspace? Please verify with a Logger or by looking at its feature cache. The AttributeKeeper or AttributeRemover are probably better at removing lists.

mark2atsafe
Rank
Userlevel 4
Badge +25

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

mark2atsafe
Rank
Userlevel 4
Badge +25

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

I will say, that I'm not sure it was working for me at first! But I can't recall if it really wasn't working, or I just wasn't looking at the results correctly. I can say definitively that at the time I posted the workspace above, it was working how I'd expect.

B
Badge +3

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

Thanks @mark2atsafe and @danatsafe. So it sounds like you don't quite expect what I'm seeing as well. I'll try to get a reasonable sample workspace up that demonstrates what seeing, my production workspace is way too big!

With AttributeManager, you seem to only get an option to remove the List Attributes rather than an option to just select the List and remove it.

However, in my case, I removed all the List attributes in AttributeManager. Sending these results to Data Inspector, neither the List, nor the Attributes show in the Features which makes it "seems" like it worked. However, on the very next Transformer the entire List, with all of its Attributes, magically reappear in Data Inspector.

Part of its my own inexperience I guess. I'm quickly realising that FME doesn't really "delete" intermediate attributes in all Transformers, even if the user has set them to be "Removed" in AttributeManager. I would hazard a guess that certain Transformer Bulk Modes just clone the data and instead only remove the references to the Attributes rather than remove the Attribute data itself? This would explain why I see the Lists completely disappear after AttributeManager in the Data Inspector results and then reappear as Exposed attributes in the next Transformer's Data Inspector results.

This is important for workspace tuning as the presence of "Hidden", unExposed lists in the data can cause significant performance degradation in downstream Transformers, so for the moment I am using AttributeKeepers to make sure the intermediate Features get split into their own schema without the List(s). It's slower to run than AttributeManager, but it speeds up the downstream Transformers that no longer have to deal with unexposed "hidden" Lists.

takashi
Rank
Userlevel 2
Badge +17

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

Hi @bwn, I've never be able to reproduce the problem you described so far. Please post a simplified workspace (and minimal data if required) that reproduces the problem.

B
Badge +3

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

@mark2atsafe and @takashi.

  1. I've attached a sample workspace that demonstrates it: sampleattributemanagernotremovinglists.fmw. There was even some weirdness on this. With Feature Caching On, and the first time I did partial runs, the AttributeManager Output Port in Data Inspector "removed" the List and List Attributes and I had to make them "reappear" in a later Inspector by sending them to another downstream processing Transformer like TopologyBuilder first, however on subsequent runs it did not do this. The issue manifested straight on the Output port of the AttributeManager and I didn't need a later Transformer. I think it demonstrates the issue either way though: The List and List Attributes are still in the Inspector results despite being set to be "Removed" in AttributeManager.
  2. Mark, I tried your sample workspace and the AttributeManager behaved differently. Unlike my attached workspace it did behave as expected. Got me stumped because in yours you have used AttributeManager to basically do exactly what my sample does and yet the outcome is different. I even modified yours to remove all List Attributes at the first AttributeManager step but yours still "behaved" correctly.

takashi
Rank
Userlevel 2
Badge +17

I think the issue is that there's a difference between the list itself and the list attributes.

The AttributeRemover has separate options for removing a list and removing attributes. If you remove the list attributes then the list still exists, so you need to remove the actual list to get rid of it fully. I suspect that the AttributeManager remove option is just removing attributes, not removing a list, hence the issue.

I think it also depends on how the list was built. I built one with an Aggregator using "All Attributes" to add to the list. This includes all FME and format attributes such as fme_color, which isn't exposed by default. So if I delete parkList{}.ParkName in the AttributeManager, then I'll still have the parkList{}.fme_color etc in that list.

But parkList{}.ParkName should be gone. If you're saying that you deleted that (or whatever your attributes are called) in the AttributeManager, but it appeared in later data, then I think that's an error.

Here's my workspace (2019.2) in case you want to give it a try: listremovalmachine.fmwt

Hopefully you'll see the same results as I do.

@bwn, thanks for sharing the worksapce. I was able to reproduce the symptom with FME 2018.1.2, 2019.1.3.1 and 2019.2.2.

In my quick test, it seems that the AttributeManager won't remove a list attribute if the feature has come from a FeatureJointer. If you replaced the FeatureJointer with a FeatureMerger, the problem would disappear.

I created a simpler workspace that demonstrates the symptom. Can you please take a closer look at this? @mark2atsafe @danatsafe

b18590-attributemanager-not-removing-lists.fmw (FME 2018.1.2)

b19817-attributemanager-not-removing-lists.fmw (FME 2019.2.2)

B
Badge +3

@bwn, thanks for sharing the worksapce. I was able to reproduce the symptom with FME 2018.1.2, 2019.1.3.1 and 2019.2.2.

In my quick test, it seems that the AttributeManager won't remove a list attribute if the feature has come from a FeatureJointer. If you replaced the FeatureJointer with a FeatureMerger, the problem would disappear.

I created a simpler workspace that demonstrates the symptom. Can you please take a closer look at this? @mark2atsafe @danatsafe

b18590-attributemanager-not-removing-lists.fmw (FME 2018.1.2)

b19817-attributemanager-not-removing-lists.fmw (FME 2019.2.2)

Thanks @takashi , so I guess that confirms a type of bug! I'll have a look through my production workspace, but yes I would suspect quite a few of the affected AttributeManagers are deriving data from FeatureJoiners output ports.

I hardly ever use FeatureMerger these days as the performance of FeatureJoiner is orders of magnitude faster in performance. Probably the speedier workaround for the moment might be trialling using an extra AttributeRemover pre-processor in the affected Transformer paths. AttributeRemover seems to both work in with a bulk mode and remove the List(s).

I had similarly thought to trial AttributeExposer as a way of "checking" if the Lists were still there but just set to be unexposed, and your sample workspace seems to confirm that.

mark2atsafe
Rank
Userlevel 4
Badge +25

Thanks @takashi , so I guess that confirms a type of bug! I'll have a look through my production workspace, but yes I would suspect quite a few of the affected AttributeManagers are deriving data from FeatureJoiners output ports.

I hardly ever use FeatureMerger these days as the performance of FeatureJoiner is orders of magnitude faster in performance. Probably the speedier workaround for the moment might be trialling using an extra AttributeRemover pre-processor in the affected Transformer paths. AttributeRemover seems to both work in with a bulk mode and remove the List(s).

I had similarly thought to trial AttributeExposer as a way of "checking" if the Lists were still there but just set to be unexposed, and your sample workspace seems to confirm that.

Thanks for hashing this out. I'll take a look at the workspace to confirm it myself and pass it on to development for a fix. I imagine it's something we'll want to sort out sooner rather than later.

mark2atsafe
Rank
Userlevel 4
Badge +25

Thanks @takashi , so I guess that confirms a type of bug! I'll have a look through my production workspace, but yes I would suspect quite a few of the affected AttributeManagers are deriving data from FeatureJoiners output ports.

I hardly ever use FeatureMerger these days as the performance of FeatureJoiner is orders of magnitude faster in performance. Probably the speedier workaround for the moment might be trialling using an extra AttributeRemover pre-processor in the affected Transformer paths. AttributeRemover seems to both work in with a bulk mode and remove the List(s).

I had similarly thought to trial AttributeExposer as a way of "checking" if the Lists were still there but just set to be unexposed, and your sample workspace seems to confirm that.

Filed as FMEENGINE-62807

takashi
Rank
Userlevel 2
Badge +17

Filed as FMEENGINE-62807

@mark2atsafe, thanks for filing the PR.

B
Badge +3

Filed as FMEENGINE-62807

Thanks @mark2atsafe, appreciated.

mark2atsafe
Rank
Userlevel 4
Badge +25

Thanks @mark2atsafe, appreciated.

@bwn and @takashi - it's marked as fixed in FME2020, build 20167 or greater. I'll see if they think it can be added to 2019.2.3 (which comes out in mid-February).

mark2atsafe
Rank
Userlevel 4
Badge +25

@bwn and @takashi - it's marked as fixed in FME2020, build 20167 or greater. I'll see if they think it can be added to 2019.2.3 (which comes out in mid-February).

Actually I'm not going to ask for a 2019 fix. Since it's a problem with the canvas and not really an issue in the translation results, and because I think it's a fairly rare case, it's probably not worth the developer's time to do that. If you disagree then please let me know and I will ask.

takashi
Rank
Userlevel 2
Badge +17

Actually I'm not going to ask for a 2019 fix. Since it's a problem with the canvas and not really an issue in the translation results, and because I think it's a fairly rare case, it's probably not worth the developer's time to do that. If you disagree then please let me know and I will ask.

Hi @mark2atsafe, it's not a problem with the Canvas. Please closer look at my demo again, which demonstrates that the AttributeManager in FME 2019.2 won't remove list attributes in some cases.

b19817-attributemanager-not-removing-lists.fmw

 

takashi
Rank
Userlevel 2
Badge +17

Filed as FMEENGINE-62807

However I agree that the severity on this issue would not be set to highest since inserting AttributeRemover is an effective workaround for this issue, if there were more critical bugs to be fixed urgently.

Reply


Community Stats

28,714
Posts
109,490
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.