Two input port ordered by group asynchronously

This is an interesting one. Let's imagine we had you indicate that PortB's input was ordered by groups, but not PortAs. Could we assume then that as soon as PortB changed, enough input had arrived in PortA that we could safely process all of the current data in PortB? It seems like we'd have to then guard against more input for that now-processed group later arriving in PortA.

 

 

Could you provide a scenario where this would give a processing benefit?

I've hit this limitation myself. Consider:

 

 

Currently It functions thus:

 

 

Port A........Port B

 

Group 1..............

 

................Group 1

 

Group 2..............

 

................Group 2

 

 

But what I have wanted to do in the past is either this:

 

 

Port A........Port B

 

Group 1..............

 

Group 2..............

 

<etc>..............

 

................Group 1

 

................Group 2

 

................<etc>

In this case I want to allow all of Port A in (Ordered) before Port B arrives (also Ordered). Obviously this will require sticking the contents of Port A into RAM until B starts arriving.

Another example (the character limit on replies is annoying)

 

 

Port A........Port B

 

Group 1..............

 

Group 2..............

 

................Group 1

 

Group 2 ..............

 

................Group 1

 

Group 2 ..............

 

................Group 2

 

................Group 2

 

 

In this case, both inputs are ordered, but again, they arrive in a different order. Port B always arrives after Port A, but it could arrive while Port A is still getting some inputs (albeit later groups).

Both Port A and Port B would be ordered (in the same order).

 

 

Consider an example with the NeighborFinder. If we had 100 groups where there are 1000 base features and 10 candidate features in each group. All 101000 features would have to arrive and be retained in memory before any processing could begin.

 

 

If on the other hand processing begins when both base and candidate features have changed, in the best case scenario only 1012 features would be in memory. In a more likely scenario all of the candidate features would probably arrive before the bases finished arriving. In that case we would still only have 2001 features in memory before processing could begin.

 

I just found this behaviour and so asked our documentation team to document it better, because it really isn't clear. In a two-port transformer you need to do Port 1/Group A, Port 2/Group A, Port 1/Group B, Port 2/Group B, Port 1/Group C, etc - at least that will be made clear now, although it is nearly impossible to organize a workspace to do that. Throw in a Candidates First type parameter and the issues are even greater. Think we need to perhaps improve the ability to order features in a workspace. Anyway, just mentioning it.

 

 

Agreed, this is impossible with two independent streams of data. The only way I can see it working is if you send both streams to a sorter and then use a tester to split them back into the two streams. But then the sorter holds all the features in memory and eats all the gains of having the input ordered by group in the later transformer.