Skip to main content
Question

FeatureHolder behaviour with disabled links or transformers upstream

mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Contributor

I use FeatureHolder to control the dataflow between FeatureWriters in order avoid blocking situations because the FeatureWriters write to the same database. So far so good.

Confusion started when the FeatureHolder seemed to let features pass unexpectedly.

Turns out the reason for this were disabled connections upstream during various trials in testing phase.

EDIT March 26: the italic writing was the original text but is NOT correct.

Solution to get predictable behaviour was to disable transformers instead of connections.

Conclusion: A disabled transformer upstream of a FeatureHolder will cause the FeatureHolder to stop everything, which is what I expected.

A disabled connection upstream of a FeatureHolder (no matter how far away) will cause the FeatureHolder to let everything through, which leads to uncontrollable behaviour from the testers point of view.

Any objections to these findings ? Is this intended behaviour, or is it a bug ?

EDIT: March 26: new revised text

FeatureHolder is not usable for synchronizing FeatureWriters, as soon as any transformer or connections upstream are disabled. In that case it will let all the features from the remaining active connections pass and therefore fail to synchronize the FeatureWriters.

Sorry for the confusion started here ... FME can be overwhelming sometimes

Michael

6 replies

erik_jan
Contributor
Forum|alt.badge.img+17
  • erik_janContributor
  • Contributor
  • March 19, 2020

Thanks @mhab Good and useful info.

mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • March 20, 2020

Can you share a screenshot of your workspace, @mhab - as I can't see to recreate this here.

In FME2020, I have this workspace:

The decelerator slows things down so I can see the flow of features. Even with a disabled connection, nothing exits the FeatureHolder until all the features are present.

Am I recreating your scenario properly? Or do I have something different? I know you have FeatureWriters, but I wanted to see if I could get it to fail in a basic configuration like this first.
mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Author
  • Contributor
  • March 20, 2020
mark2atsafe wrote:

Can you share a screenshot of your workspace, @mhab - as I can't see to recreate this here.

In FME2020, I have this workspace:

The decelerator slows things down so I can see the flow of features. Even with a disabled connection, nothing exits the FeatureHolder until all the features are present.

Am I recreating your scenario properly? Or do I have something different? I know you have FeatureWriters, but I wanted to see if I could get it to fail in a basic configuration like this first.

Thanks Mark for following up on this.

Difference in my case is it must be different inputs entering into FeatureHolder.

Custom Transformer: StreamHolder

Here the full workspace (as uploaded).

StreamHolder_Test.fmw

mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • March 25, 2020
mhab wrote:

Thanks Mark for following up on this.

Difference in my case is it must be different inputs entering into FeatureHolder.

Custom Transformer: StreamHolder

Here the full workspace (as uploaded).

StreamHolder_Test.fmw

So, I really can't see anything wrong happening here! Maybe it's my computer is different to yours in some way, or my version of FME (I'm trying on 2020).

The one thing I will mention is that sometimes it looks like features are suddenly jumping through, but I think that's due to the lag in the feature counts being updated. Like Feature 4 looks like it jumps in straight after feature 3, without the delay; but what I think happens is that the delay is for feature 4, and we're just not seeing feature 3's feature count until later (when feature 4 is released by the Decelerator, then the cache of log messages is flushed, so to speak).

Anyway, I put a small movie demo together, here: https://www.screencast.com/t/89Lv8SNf

So now you can see what I can see! If your workspace looks different to this when it runs, I'd be very interested for sure. But I just don't see features leaking through the FeatureHolder until Feature 4 arrives, whatever I do.
mhab
Contributor
Forum|alt.badge.img+10
  • mhabContributor
  • Author
  • Contributor
  • March 26, 2020
mark2atsafe wrote:

So, I really can't see anything wrong happening here! Maybe it's my computer is different to yours in some way, or my version of FME (I'm trying on 2020).

The one thing I will mention is that sometimes it looks like features are suddenly jumping through, but I think that's due to the lag in the feature counts being updated. Like Feature 4 looks like it jumps in straight after feature 3, without the delay; but what I think happens is that the delay is for feature 4, and we're just not seeing feature 3's feature count until later (when feature 4 is released by the Decelerator, then the cache of log messages is flushed, so to speak).

Anyway, I put a small movie demo together, here: https://www.screencast.com/t/89Lv8SNf

So now you can see what I can see! If your workspace looks different to this when it runs, I'd be very interested for sure. But I just don't see features leaking through the FeatureHolder until Feature 4 arrives, whatever I do.

Wow, Mark thanks for going all the way with making a video about this.

I start to understand that I expect behaviour from FeatureHolder which isn't supposed to work.

As I started to prepare screenshots to explain what I mean, I fell into doubts about the whole discussion I started here.

My uploaded test workspace does behave as you describe in the video and it is the same behaviour in FME 2018, 2019 and 2020.

I can't repeat the differences I mentioned with disabling connections vs. disabling transformers.

I will do some more testing and may have to edit my original text. I'm sorry but somewhere I got lost ...

mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • March 26, 2020
mhab wrote:

Wow, Mark thanks for going all the way with making a video about this.

I start to understand that I expect behaviour from FeatureHolder which isn't supposed to work.

As I started to prepare screenshots to explain what I mean, I fell into doubts about the whole discussion I started here.

My uploaded test workspace does behave as you describe in the video and it is the same behaviour in FME 2018, 2019 and 2020.

I can't repeat the differences I mentioned with disabling connections vs. disabling transformers.

I will do some more testing and may have to edit my original text. I'm sorry but somewhere I got lost ...

That's OK. Not a problem. The number of times I've filed a report with our developers, and then found out that I was the issue, rather than FME. Ouch. If you do find something to investigate, please do post back. The other cause might have been caching, which I meant to mention can affect what you see on screen (again, I speak from personal experience!)

Reply


Helpful Members This Week

Community Stats

31,197
Posts
118,251
Replies
38,801
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.