Question

ListBasedFeatureMerger


Badge +10

I've never used the ListBasedFeatureMerger before, the documentation suggests it might be what I want but it isn't working as I thought

I'd assumed that if the Requestor had a list that looked like

list{0} = apple

list{1} = banana

list{2} = cherry

and the supplier had an attribute with value banana there would be an output via the merged port. Is this not correct?


16 replies

Userlevel 4
Badge +25

Would it need to be list{0}.value = apple, list{1}.value=banana, etc? I haven't tried it, but my first thought is that it might have to be a specific list attribute, not just a list without the attribute (ie it needs the .value part, whatever it is called)

Badge +10

Would it need to be list{0}.value = apple, list{1}.value=banana, etc? I haven't tried it, but my first thought is that it might have to be a specific list attribute, not just a list without the attribute (ie it needs the .value part, whatever it is called)

I was originally trying it with a list created by the ShortestPathFinder, so the list was routes{}.uniqueid

 

 

Userlevel 4
Badge +25
I was originally trying it with a list created by the ShortestPathFinder, so the list was routes{}.uniqueid

 

 

Then it's not that. OK. I'll have a deeper look, but from reading the documentation my understanding matches yours.

Badge +10

It's very strange, I've managed to get a merged feature out after creating a simple list, but no luck when trying with real data when I know the value exists

Userlevel 4
Badge +25

So I made up an example and it worked fine for me. Can you confirm this workspace works for you?

listbasedfeaturemerger.fmwt

Just wondering if you can get any example to work, or whether it's just the one specific scenario.

Userlevel 4
Badge +25

So I made up an example and it worked fine for me. Can you confirm this workspace works for you?

listbasedfeaturemerger.fmwt

Just wondering if you can get any example to work, or whether it's just the one specific scenario.

OK, I see you already answered my question. Maybe a space character suffixed to one set of values? Can you put them through a trimmer? Awkward when it's a list I know, but it seems the most obvious issue.

Badge +10

OK, I see you already answered my question. Maybe a space character suffixed to one set of values? Can you put them through a trimmer? Awkward when it's a list I know, but it seems the most obvious issue.

It definitely seems like there's something at the beginning or end.

If I use a listsearcher with a value of same, i get a match, but using ^same$ gives no match

If I explode the list however, and test for regex ^same$ I get 72 matches

I can actually search for the multiple values I want in the list in python if needed, but at the moment that means either hardcoding a list within the python caller (which is super quick but not easily updateable) or dynamically creating the list and attaching it to every feature and then comparing for matches between the lists.

It seemed like the Listbasedfeaturemerger might have been an easier option but perhaps not!

(I'm not actually testing for the _direction list attribute, as there are only two values this would be straightforward anyway, it just seemed that as this is directly created by the shortestpathfinder transformer it was the one least subject to any errors in data for testing.)

Badge +10

listbasedfeaturemerger.fmwt

Sample attached of no output via the listbasedfeaturemerger despite the value being present in the list

Userlevel 2
Badge +17

In order to merge Suppliers' attributes to a Requestor feature with the ListBasedFeatureMerger, every element in the Requestor's list should be supplied by the Supplier features. In your example, there are two distinct values in the list - 'same' and 'opposite'. so at least two Supplier features should be supplied; one should have 'same' and another should have 'opposite' as the join key attribute.

See also this example: listbasedfeaturemerger-2.fmwt (FME 2018.1.0.3)

Badge +10

In order to merge Suppliers' attributes to a Requestor feature with the ListBasedFeatureMerger, every element in the Requestor's list should be supplied by the Supplier features. In your example, there are two distinct values in the list - 'same' and 'opposite'. so at least two Supplier features should be supplied; one should have 'same' and another should have 'opposite' as the join key attribute.

See also this example: listbasedfeaturemerger-2.fmwt (FME 2018.1.0.3)

If it's the case that "every element in the Requestor's list should be supplied by the Supplier features" then this transformer isn't going to do what I want.

Although this line in the help "The Incomplete Requestors parameter specifies whether or not to merge Suppliers onto partially matching Requestors when there are keys in the Requestor List Attribute that have no matching Suppliers." suggests that features can be output via the merged port even if every list value in the requestor doesn't have a corresponding supplier value.

I have a pythoncaller set up to do what I need to anyway now - which is to only keep features where at least one feature in the list matches at least one value in a supplied list. This transformer just looked like it would do the same thing and I prefer to use native transformers instead of python where possible.

 

Userlevel 4
Badge +25

If it's the case that "every element in the Requestor's list should be supplied by the Supplier features" then this transformer isn't going to do what I want.

Although this line in the help "The Incomplete Requestors parameter specifies whether or not to merge Suppliers onto partially matching Requestors when there are keys in the Requestor List Attribute that have no matching Suppliers." suggests that features can be output via the merged port even if every list value in the requestor doesn't have a corresponding supplier value.

I have a pythoncaller set up to do what I need to anyway now - which is to only keep features where at least one feature in the list matches at least one value in a supplied list. This transformer just looked like it would do the same thing and I prefer to use native transformers instead of python where possible.

 

It looks to me like the content does get merged, it's just that the feature comes out of the UnmergedRequestor port. eg here there is no neighborhood in Vancouver called MarksVille (shame) but the other data is still merged for neighborhoods that do exist:

Notice how the zzzz list has no entry for 0, because xxxx{0} has no match. If MarksVille were in xxxx{4} then zzzz{4} wouldn't exist.

I can sort of see why it was designed to do this. I mean, some parts are merged, but some are not. So there's an argument that it could count as Merged or Unmerged.

Anyway, if that's how it works for you too, you could always check for the existence of the zzzz list. Use a ListElementCounter on the Unmerged output and if it returns zero then it's totally Unmatched. If it is greater than zero then it's partially matched.

I haven't tried your sample workspace, but I'll give it a go now.

Userlevel 4
Badge +25

It looks to me like the content does get merged, it's just that the feature comes out of the UnmergedRequestor port. eg here there is no neighborhood in Vancouver called MarksVille (shame) but the other data is still merged for neighborhoods that do exist:

Notice how the zzzz list has no entry for 0, because xxxx{0} has no match. If MarksVille were in xxxx{4} then zzzz{4} wouldn't exist.

I can sort of see why it was designed to do this. I mean, some parts are merged, but some are not. So there's an argument that it could count as Merged or Unmerged.

Anyway, if that's how it works for you too, you could always check for the existence of the zzzz list. Use a ListElementCounter on the Unmerged output and if it returns zero then it's totally Unmatched. If it is greater than zero then it's partially matched.

I haven't tried your sample workspace, but I'll give it a go now.

OK, it means you need to generate a list of matches in the output, which I notice you aren't doing; but then you can check if anything at all was a match and pass/fail as necessary.

Userlevel 2
Badge +17

If it's the case that "every element in the Requestor's list should be supplied by the Supplier features" then this transformer isn't going to do what I want.

Although this line in the help "The Incomplete Requestors parameter specifies whether or not to merge Suppliers onto partially matching Requestors when there are keys in the Requestor List Attribute that have no matching Suppliers." suggests that features can be output via the merged port even if every list value in the requestor doesn't have a corresponding supplier value.

I have a pythoncaller set up to do what I need to anyway now - which is to only keep features where at least one feature in the list matches at least one value in a supplied list. This transformer just looked like it would do the same thing and I prefer to use native transformers instead of python where possible.

 

The description you have quoted doesn't mention that the incomplete requestor will be output from which port. The Output Posts section in the help describes the behavior more clearly.

"Output Ports

Merged

Requestors that have matching Suppliers for all of the keys in the specified Requestor List AttributeOnly fully matched Requestors are output onto this port.

UnmergedRequestor

This port used to be labelled as Incomplete and has retained the semantics of that label. So, this port label is a shorthand for 'NotFullyMerged'. Partially matched Requestors are output onto this port, and all matching Suppliers are merged onto the output features if Incomplete Requestors is set to Merge Supplier Information. If a key in the Requestor List Attribute is null, that key will not match any Suppliers, and as a result, the Requestor feature will be output onto this port. For example, if the Requestor keys are:

list{0}.key = 1

list{1}.key = <null>

list{2}.key = 3

and furthermore there are Suppliers with 'supplierKey' values of 1, <null>, and 3, and Incomplete Requestors is set to Merge Supplier Information, then the Requestor will be output onto the UnmergedRequestor port, with Suppliers merged onto it whose 'supplierKey' is set to 1 or 3."

Badge +10

The description you have quoted doesn't mention that the incomplete requestor will be output from which port. The Output Posts section in the help describes the behavior more clearly.

"Output Ports

Merged

Requestors that have matching Suppliers for all of the keys in the specified Requestor List AttributeOnly fully matched Requestors are output onto this port.

UnmergedRequestor

This port used to be labelled as Incomplete and has retained the semantics of that label. So, this port label is a shorthand for 'NotFullyMerged'. Partially matched Requestors are output onto this port, and all matching Suppliers are merged onto the output features if Incomplete Requestors is set to Merge Supplier Information. If a key in the Requestor List Attribute is null, that key will not match any Suppliers, and as a result, the Requestor feature will be output onto this port. For example, if the Requestor keys are:

list{0}.key = 1

list{1}.key = <null>

list{2}.key = 3

and furthermore there are Suppliers with 'supplierKey' values of 1, <null>, and 3, and Incomplete Requestors is set to Merge Supplier Information, then the Requestor will be output onto the UnmergedRequestor port, with Suppliers merged onto it whose 'supplierKey' is set to 1 or 3."

Hadn't spotted that, I'd made an assumption that unmerged was, well unmerged, rather than incomplete :-)

 

Interestingly, if the option to generate a list is turned on, this isn't exposed via the UnmergedRequestor port even though it is created.

Userlevel 4
Badge +25

So it seems that this behaviour is documented, but just not very well.

This port used to be labelled as Incomplete and has retained the semantics of that label. So, this port label is a shorthand for 'NotFullyMerged'. Partially matched Requestors are output onto this port, and all matching Suppliers are merged onto the output features if Incomplete Requestors is set to Merge Supplier Information. If a key in the Requestor List Attribute is null, that key will not match any Suppliers, and as a result, the Requestor feature will be output onto this port. For example, if the Requestor keys are:list{0}.key = 1list{1}.key = <null>list{2}.key = 3and furthermore there are Suppliers with 'supplierKey' values of 1, <null>, and 3, and Incomplete Requestors is set to Merge Supplier Information, then the Requestor will be output onto the UnmergedRequestor port, with Suppliers merged onto it whose 'supplierKey' is set to 1 or 3.The transformer also doesn't carry over exposed attributes to the output, even for partial matches. I looked it up and there is already an enhancement request for improved behaviour. fyi the reference number is FMEENGINE-28521. I'll post a link to this thread and see if I can increase the priority some, so no-one else suffers the same issue.
Userlevel 4
Badge +25

Hadn't spotted that, I'd made an assumption that unmerged was, well unmerged, rather than incomplete :-)

 

Interestingly, if the option to generate a list is turned on, this isn't exposed via the UnmergedRequestor port even though it is created.

Right, there seem to be one or two flaws in both the design and the documentation here. I've pushed the enhancement request back to the developers for consideration. I know that it's not a commonly used transformer, but if you, me, and Takashi are all having problems, then there must be something worth looking at.

Reply