Solved

JSON templater doesn't wait for sub-templates

V
Badge

I'm facing a strange situation in v2017.1

 

I have a fairly complicated JSON Templater, using several sub-templates, joining several inputs on a "group sub-features by"

The output have most sub-templates outed as "None", and as I watched it processing, I realized the JSON Tempater started outputting some features even before receiving a single input from some of the sub-templates inputs streams. (those are quite longer to process, retrieving several additional data from the database, then using python to pivot some data and generate list of json elements itself to nest in the main json)

 

How is that even possible ? Is it not waiting for everything being provided to it like any other junction transformers ?
icon

Best answer by david_r 24 May 2018, 16:18

View original

16 replies

david_r
Rank
Userlevel 4

Have you maybe set "group sub-features are consecutive" to Yes?

V
Badge

Have you maybe set "group sub-features are consecutive" to Yes?

Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

david_r
Rank
Userlevel 4
Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

Have you tried setting it to "No"?
V
Badge
Yes, They are. And that's part of the problem, the features that are "late" are still piling up in a sorter while the json templater is rushing and assigning nones while he should wait for them.

 

 

I thought I had a sorting issue but the jsonTemplater didn't received ANY inputs from that sub-template stream at the time it starts to produce outputs.

 

Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

david_r
Rank
Userlevel 4
Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

It's not a bug, it's by design. If you activate that setting you're basically telling the transformer that you guarantee that all the group-by values will be input pre-sorted by that value.

 

From the documentation:

 

The behavior of the transformer is undefined if the features in a group do not all arrive

 

consecutively. It is likely that some sub-features will not be processed if this happens.

 

V
Badge
Yes, that works, but that would be really bad for the performance of the process.... and that doesn't explain why it doesn't bother for waiting for the input... is that a known bug ? corrected in v2018 ?

 

But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

david_r
Rank
Userlevel 4
But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

Does all the features entering all of the input ports of the JsonTemplater have this group-by attribue? Or could it be that only the sub-templates have this attribute?
V
Badge
But... they are pre-sorted. they are just not there yet because they are (temporarily) retained by a sorter (or an aggregator) and those branches are slower.

 

 

I mean I had read that documentation, "if the features in a group do not all arrive" should mean that the json templater either

 

  • received an input through that sub-template's stream that is "greater" (value-of-grouping-attribute-sorting-wise) than the grouping attribute value of the root feature (thus it knows that there is nothing to process for the current grouping attribute value)
  • or received a "termination signal" from that sub-template stream to ensure that there is nothing to wait for. In my case it didn't even started.
There is no way FME would by design intend to have different branch of the same process race against each other with random output based on which branch is completed first...

 

Yes, every single sub-template input stream posess the group-by attribute

 

 

david_r
Rank
Userlevel 4
Yes, every single sub-template input stream posess the group-by attribute

 

 

I suspect the ROOT template trigger features will also need this group-by attribute

 

 

V
Badge
Yes, every single sub-template input stream posess the group-by attribute

 

 

Of course, it does have it too

 

 

david_r
Rank
Userlevel 4
Of course, it does have it too

 

 

I suspect the best way forward is if you're able to create a minimal workspace that demonstrates the issue and post it here, it's getting difficult to guess :-)
V
Badge
Of course, it does have it too

 

 

I will try to create one, not easy without embedding data.

 

david_r
Rank
Userlevel 4
I will try to create one, not easy without embedding data.

 

If your data isn't sensitive, look into using the Player and the Recorder transformers to isolate just a part of the workspace in question, without needing the full data sources.

 

If your data is sensitive, consider sending it to Safe support rather than posting it here.
V
Badge
I will try to create one, not easy without embedding data.

 

none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

david_r
Rank
Userlevel 4
none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

If it's not possible to reproduce the issue with a simpler workspace, I'm going to go out on a limb and say it's most probably something wrong with the original workspace...

 

Have you tried setting break points on all the input ports of the JsonTemplater to check that the group-by attribute is always there, and that all the features are indeed sorted by the group-by attribute?
V
Badge
none2none.fmw I don't manage to reproduce it on a simpler case, it's basically this (except the python creator and first python caller should be SQLServer queries) but this one works as intended, there is nothing outputted by the jsonTemplater until the sorter pours out too...

 

 

Yeah, I'm rethinkling my sorting/aggregating from scratch and simplyfiyng the process, I will add to this post if it still fail.

 

 

Thanks for your help, anyway ;)

 

 

Reply


Community Stats

28,661
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.