Solved

XML flattening of lists with one entry


Userlevel 1
Badge +22

Hi all.

I'm having a hard time figuring out why lists with one entry, especially if it's read from XML, is mishandled in FME.

A list may have any number of entries, and if reading such from XML, they often have any number including zero entries.

When flattening the output, cycling thru the list, one normally uses ListExploder. However, if the list in question has one entry, it's auto-exploded, which makes ListExploder reject it. So I have to check the rejected features to see, if the list had been auto-exploded or if it had no entries at all. E.g. two or more transformers instead of just one.

Why?

Wouldn't it be more logical to _always_ create a list, if there are 1 or more entries ?

It works as such in 2021.0

Is this behaviour specific to XML (or JSON?) flattening ?

Cheers.

icon

Best answer by nielsgerrits 22 June 2021, 13:42

View original

12 replies

Userlevel 1
Badge +22

A small example: An XML with 5 lists (container tags) with 0-4 entries. Lists with 0 or 1 entries gets rejected when read.

Userlevel 6
Badge +33

In the XMLFlattener Parameters, click Flatten Options..., Click Advanced... then change cardinality from

cardinality="+{?}"

to

cardinality="{}"

Also see this question.

Userlevel 1
Badge +22

In the XMLFlattener Parameters, click Flatten Options..., Click Advanced... then change cardinality from

cardinality="+{?}"

to

cardinality="{}"

Also see this question.

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

Userlevel 1
Badge +22

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

And why is this behaviour not the default ??

Userlevel 6
Badge +33

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

To be honest I only know this because I know the other question exists. I need to look it up every time I want to fix it. About documentation, takashi points to this piece. But I never went through it because setting cardinality="{}" is all I need.

 

And yes, I think this should be the default, because with the current default the result is less predictable.

Userlevel 1
Badge +22

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

Alas, it worked in my small test, but not in my original workspace.

The difference is that the original workspace use old-school readers, not FeatureReader.

I'll see if replacing the reader with a FeatureReader will have an effect.

Userlevel 1
Badge +22

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

Nope. The problem was another.

It seems that changing the "cardinality" makes every single XML tag into a list of its subtags, regardless of whether or not it's warranted.

So instead of rootTag.subTag.listTag{} it's rootTag{0}.subTag{0}.listTag{}.

I assume that this is because the XML reader is oblivious to any schemas and structure ?

--

@nielsgerrits​ , are you aware of any documentation of the possible values for "cardinality"?

Or is it trial-and-error for now?

Cheers

Userlevel 6
Badge +33

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

Have you read this one? This is all I have.

Userlevel 1
Badge +22

Thanks Niels.

Wow. That's probably the simplest and most easy to use, yet at the same time the most convoluted and intricate, answer I've seen in a very long while.

It works, but where in the world did you find the documentation on that ???

Are there any official documentation on all such "advanced" setups possibly in FME ?

And what does "cardinality" even mean in this case ?

 

Yes, I found the very same documentation. It concerns the elusive xfMap, of which I would love a webinar/course to master.

One can use "cardinality" to specify what important tags are singulars, and what are arrays, and the below works for my data:

cardinality="*/DelLedningItems/DelLedning{}/KnaekpunktItems/Knaekpunkt{}/* +{?}"

My input data looks sort of like this, extracting "LedningGroup/Ledning" from the XML:

<LedningGroup xmlns="http://www.danva.dk/xml/schemas/dandas/20120102">
<Ledning OpstroemKnudenavn="A" NedstroemKnudenavn="C" DobbeltlednNr="">
  <DelLedningItems>
    <DelLedning OpstroemKnudenavn="A" NedstroemKnudenavn="B">
      <KnaekpunktItems>
        <Knaekpunkt Sortering="1">
          <XKoordinat>0000</XKoordinat>
          <YKoordinat>0000</YKoordinat>
        </Knaekpunkt>
        <Knaekpunkt Sortering="2">
          <XKoordinat>0000</XKoordinat>
          <YKoordinat>0000</YKoordinat>
        </Knaekpunkt>
      </KnaekpunktItems>
      <LabelDelledningItems />
    </DelLedning>
    <DelLedning OpstroemKnudenavn="B" NedstroemKnudenavn="C">
      <KnaekpunktItems>
        <Knaekpunkt Sortering="1">
          <XKoordinat>0000</XKoordinat>
          <YKoordinat>0000</YKoordinat>
        </Knaekpunkt>
        <Knaekpunkt Sortering="2">
          <XKoordinat>0000</XKoordinat>
          <YKoordinat>0000</YKoordinat>
        </Knaekpunkt>
      </KnaekpunktItems>
      <LabelDelledningItems />
    </DelLedning>
  </DelLedningItems>
</Ledning>
</LedningGroup>

The first setting gets me all "Knaekpunkt" with 1 or more items as a list.

Userlevel 3
Badge +13

Hello @Lars I Nielsen​, list are sort of funny in FME. If there is only one element, we default to read it as a simple datatype, such as string or number. When lists are sent to the XMLFragmenter or XMLFlattener, for example, FME interprets the feature path as is and only knows to make a list if it gets multiple values - schemaless. However, if a list is reflected in a schema, and we know the schema, it will be read properly as a list feature. Hope this helps, Kailin.

Userlevel 1
Badge +22

Hello @Lars I Nielsen​, list are sort of funny in FME. If there is only one element, we default to read it as a simple datatype, such as string or number. When lists are sent to the XMLFragmenter or XMLFlattener, for example, FME interprets the feature path as is and only knows to make a list if it gets multiple values - schemaless. However, if a list is reflected in a schema, and we know the schema, it will be read properly as a list feature. Hope this helps, Kailin.

Thanks Kailin.

As you can see from my other replies, I've been borrowing into the setup for the XML parsing, especially the "cardinality" setting. I.e. into the whole xfMap setup regime. It does solve my immidiate concerns, but is a little coarse for production use.

Can you mitigate my wish for a Safe Software course or workshop into everything xfMap and more, and how to use it all ? I think Don will be pleased by this wish :-)

I have been looking towards the schema driven XML, but unfortunately the data in question here does not reference a schema document, so it wouldn't work anyway. There may be a schema document somewhere, and if I find it, I will test it forthwith. It's a proprietary Danish format for sewer utility data.

Cheers.

Userlevel 3
Badge +13

Thanks Kailin.

As you can see from my other replies, I've been borrowing into the setup for the XML parsing, especially the "cardinality" setting. I.e. into the whole xfMap setup regime. It does solve my immidiate concerns, but is a little coarse for production use.

Can you mitigate my wish for a Safe Software course or workshop into everything xfMap and more, and how to use it all ? I think Don will be pleased by this wish :-)

I have been looking towards the schema driven XML, but unfortunately the data in question here does not reference a schema document, so it wouldn't work anyway. There may be a schema document somewhere, and if I find it, I will test it forthwith. It's a proprietary Danish format for sewer utility data.

Cheers.

Hello again @Lars I Nielsen​ ,  xfMap contnet is on our roadmap and the estimated time would be Summer 2022, but this timeline may change. Best, Kailin.

Reply