Skip to main content
Question

FeatureHolder behaviour with disabled links or transformers upstream

  • March 19, 2020
  • 6 replies
  • 23 views
mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Contributor
  • 106 replies

I use FeatureHolder to control the dataflow between FeatureWriters in order avoid blocking situations because the FeatureWriters write to the same database. So far so good.

Confusion started when the FeatureHolder seemed to let features pass unexpectedly.

Turns out the reason for this were disabled connections upstream during various trials in testing phase.

EDIT March 26: the italic writing was the original text but is NOT correct.

Solution to get predictable behaviour was to disable transformers instead of connections.

Conclusion: A disabled transformer upstream of a FeatureHolder will cause the FeatureHolder to stop everything, which is what I expected.

A disabled connection upstream of a FeatureHolder (no matter how far away) will cause the FeatureHolder to let everything through, which leads to uncontrollable behaviour from the testers point of view.

Any objections to these findings ? Is this intended behaviour, or is it a bug ?

EDIT: March 26: new revised text

FeatureHolder is not usable for synchronizing FeatureWriters, as soon as any transformer or connections upstream are disabled. In that case it will let all the features from the remaining active connections pass and therefore fail to synchronize the FeatureWriters.

Sorry for the confusion started here ... FME can be overwhelming sometimes

Michael

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

erik_jan
Contributor
Forum|alt.badge.img+22
  • erik_janContributor
  • Contributor
  • 2179 replies
  • March 19, 2020

Thanks @mhab Good and useful info.

mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • March 20, 2020

Can you share a screenshot of your workspace, @mhab - as I can't see to recreate this here.

In FME2020, I have this workspace:

The decelerator slows things down so I can see the flow of features. Even with a disabled connection, nothing exits the FeatureHolder until all the features are present.

Am I recreating your scenario properly? Or do I have something different? I know you have FeatureWriters, but I wanted to see if I could get it to fail in a basic configuration like this first.
FME Evangelist to the Rich and Famous!!
mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Author
  • Contributor
  • 106 replies
  • March 20, 2020

Can you share a screenshot of your workspace, @mhab - as I can't see to recreate this here.

In FME2020, I have this workspace:

The decelerator slows things down so I can see the flow of features. Even with a disabled connection, nothing exits the FeatureHolder until all the features are present.

Am I recreating your scenario properly? Or do I have something different? I know you have FeatureWriters, but I wanted to see if I could get it to fail in a basic configuration like this first.

Thanks Mark for following up on this.

Difference in my case is it must be different inputs entering into FeatureHolder.

Custom Transformer: StreamHolder

Here the full workspace (as uploaded).

StreamHolder_Test.fmw

mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • March 25, 2020

Thanks Mark for following up on this.

Difference in my case is it must be different inputs entering into FeatureHolder.

Custom Transformer: StreamHolder

Here the full workspace (as uploaded).

StreamHolder_Test.fmw

So, I really can't see anything wrong happening here! Maybe it's my computer is different to yours in some way, or my version of FME (I'm trying on 2020).

The one thing I will mention is that sometimes it looks like features are suddenly jumping through, but I think that's due to the lag in the feature counts being updated. Like Feature 4 looks like it jumps in straight after feature 3, without the delay; but what I think happens is that the delay is for feature 4, and we're just not seeing feature 3's feature count until later (when feature 4 is released by the Decelerator, then the cache of log messages is flushed, so to speak).

Anyway, I put a small movie demo together, here: https://www.screencast.com/t/89Lv8SNf

So now you can see what I can see! If your workspace looks different to this when it runs, I'd be very interested for sure. But I just don't see features leaking through the FeatureHolder until Feature 4 arrives, whatever I do.
FME Evangelist to the Rich and Famous!!
mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Author
  • Contributor
  • 106 replies
  • March 26, 2020

So, I really can't see anything wrong happening here! Maybe it's my computer is different to yours in some way, or my version of FME (I'm trying on 2020).

The one thing I will mention is that sometimes it looks like features are suddenly jumping through, but I think that's due to the lag in the feature counts being updated. Like Feature 4 looks like it jumps in straight after feature 3, without the delay; but what I think happens is that the delay is for feature 4, and we're just not seeing feature 3's feature count until later (when feature 4 is released by the Decelerator, then the cache of log messages is flushed, so to speak).

Anyway, I put a small movie demo together, here: https://www.screencast.com/t/89Lv8SNf

So now you can see what I can see! If your workspace looks different to this when it runs, I'd be very interested for sure. But I just don't see features leaking through the FeatureHolder until Feature 4 arrives, whatever I do.

Wow, Mark thanks for going all the way with making a video about this.

I start to understand that I expect behaviour from FeatureHolder which isn't supposed to work.

As I started to prepare screenshots to explain what I mean, I fell into doubts about the whole discussion I started here.

My uploaded test workspace does behave as you describe in the video and it is the same behaviour in FME 2018, 2019 and 2020.

I can't repeat the differences I mentioned with disabling connections vs. disabling transformers.

I will do some more testing and may have to edit my original text. I'm sorry but somewhere I got lost ...

mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • March 26, 2020

Wow, Mark thanks for going all the way with making a video about this.

I start to understand that I expect behaviour from FeatureHolder which isn't supposed to work.

As I started to prepare screenshots to explain what I mean, I fell into doubts about the whole discussion I started here.

My uploaded test workspace does behave as you describe in the video and it is the same behaviour in FME 2018, 2019 and 2020.

I can't repeat the differences I mentioned with disabling connections vs. disabling transformers.

I will do some more testing and may have to edit my original text. I'm sorry but somewhere I got lost ...

That's OK. Not a problem. The number of times I've filed a report with our developers, and then found out that I was the issue, rather than FME. Ouch. If you do find something to investigate, please do post back. The other cause might have been caching, which I meant to mention can affect what you see on screen (again, I speak from personal experience!)

FME Evangelist to the Rich and Famous!!

Community Stats

32,076
Posts
121,797
Replies
39,707
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.