Skip to main content
Solved

JSON templater doesn't wait for sub-templates

V
Forum|alt.badge.img

I'm facing a strange situation in v2017.1

 

I have a fairly complicated JSON Templater, using several sub-templates, joining several inputs on a "group sub-features by"

The output have most sub-templates outed as "None", and as I watched it processing, I realized the JSON Tempater started outputting some features even before receiving a single input from some of the sub-templates inputs streams. (those are quite longer to process, retrieving several additional data from the database, then using python to pivot some data and generate list of json elements itself to nest in the main json)

 

How is that even possible ? Is it not waiting for everything being provided to it like any other junction transformers ?

Best answer by david_r

Have you maybe set "group sub-features are consecutive" to Yes?

View original
Did this help you find an answer to your question?

16 replies

david_r
Celebrity
  • david_rCelebrity
  • Best Answer
  • May 24, 2018

Have you maybe set "group sub-features are consecutive" to Yes?

V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
david_r wrote:

Have you maybe set "group sub-features are consecutive" to Yes?

Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

Have you tried setting it to "No"?
V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

It's not a bug, it's by design. If you activate that setting you're basically telling the transformer that you guarantee that all the group-by values will be input pre-sorted by that value.

 

From the documentation:

 

The behavior of the transformer is undefined if the features in a group do not all arrive

 

consecutively. It is likely that some sub-features will not be processed if this happens.

 

V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

Does all the features entering all of the input ports of the JsonTemplater have this group-by attribue? Or could it be that only the sub-templates have this attribute?
V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

Yes, every single sub-template input stream posess the group-by attribute

 

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
Yes, every single sub-template input stream posess the group-by attribute

 

 

I suspect the ROOT template trigger features will also need this group-by attribute

 

 

V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
Yes, every single sub-template input stream posess the group-by attribute

 

 

Of course, it does have it too

 

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
Of course, it does have it too

 

 

I suspect the best way forward is if you're able to create a minimal workspace that demonstrates the issue and post it here, it's getting difficult to guess :-)
V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
Of course, it does have it too

 

 

I will try to create one, not easy without embedding data.

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
I will try to create one, not easy without embedding data.

 

If your data isn't sensitive, look into using the Player and the Recorder transformers to isolate just a part of the workspace in question, without needing the full data sources.

 

If your data is sensitive, consider sending it to Safe support rather than posting it here.
V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 24, 2018
vchalmel wrote:
I will try to create one, not easy without embedding data.

 

none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

david_r
Celebrity
  • david_rCelebrity
  • May 24, 2018
vchalmel wrote:
none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

If it's not possible to reproduce the issue with a simpler workspace, I'm going to go out on a limb and say it's most probably something wrong with the original workspace...

 

Have you tried setting break points on all the input ports of the JsonTemplater to check that the group-by attribute is always there, and that all the features are indeed sorted by the group-by attribute?
V
Forum|alt.badge.img
  • vchalmel
  • Author
  • May 25, 2018
vchalmel wrote:
none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

Yeah, I'm rethinkling my sorting/aggregating from scratch and simplyfiyng the process, I will add to this post if it still fail.

 

 

Thanks for your help, anyway ;)

 

 

Reply


Helpful Members This Week

Community Stats

31,340
Posts
118,850
Replies
38,929
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.