Extract XML from URL

First of all, there's an invalid character on lines 59456 and 59473, e.g.:

<notes>IPID - 10762190
SS - 42109, 25614

Actuator Max 30m ? Min 15m</notes>

You might want to e.g. use the StringReplacer to fix those first.

You can then read the XML like this (or similar using the FeatureReader):

Result:

When I try to read it using a FeatureReader I get:

When I check the file in Notepad++ to line 59455 I see some newlines, probably from the source data field SiteNotes.

This corrupts the XML structure which is causing your issue.

You also can only use FME to do check the XML structure:

• Read the xml as text file, change parameter to read whole file at once (entire text in one feature). Make sure you read the correct encoding. (UTF-8).
• Connect a XMLValidator. The validator will stop at the first error.

Tip: Create a two samples of the file for faster iteration while working. One corrupt file, one good file. When it works, just change to the original big file.

Use the good file to generate the paths from the structure you need with a FeatureReader.

So you need to replace or remove the incorrect newlines in SiteNotes. One way to do this:

• Remove all correct newlines. (not the incorrect ones) StringReplacer, Replace Regular Expression, replace > by >.
• Replace all remaining newlines with another, not used, character, for example a pipe. This way you are able to restore the newlines from SiteNotes in a later phase of the proces. StringReplacer, Replace Regular Expression, replace by |.
• Connect a XMLValidator to check if this was the only error.

Now you can write the XML file or just process the corrected file.

• XMLFragmenter. (Create a feature for each logger.) Match loggers/logger
• XMLFlattener. (This can also be done in the XMLFragmenter, but for learning this might be easier to do step by step.) Match logger.

Edit: added workspace template

parsexml.fmwt

That's incredibly helpful - thank you so much for your detailed response!

When I try to read it using a FeatureReader I get:

When I check the file in Notepad++ to line 59455 I see some newlines, probably from the source data field SiteNotes.

This corrupts the XML structure which is causing your issue.

You also can only use FME to do check the XML structure:

• Read the xml as text file, change parameter to read whole file at once (entire text in one feature). Make sure you read the correct encoding. (UTF-8).
• Connect a XMLValidator. The validator will stop at the first error.

Tip: Create a two samples of the file for faster iteration while working. One corrupt file, one good file. When it works, just change to the original big file.

Use the good file to generate the paths from the structure you need with a FeatureReader.

So you need to replace or remove the incorrect newlines in SiteNotes. One way to do this:

• Remove all correct newlines. (not the incorrect ones) StringReplacer, Replace Regular Expression, replace > by >.
• Replace all remaining newlines with another, not used, character, for example a pipe. This way you are able to restore the newlines from SiteNotes in a later phase of the proces. StringReplacer, Replace Regular Expression, replace by |.
• Connect a XMLValidator to check if this was the only error.

Now you can write the XML file or just process the corrected file.

• XMLFragmenter. (Create a feature for each logger.) Match loggers/logger
• XMLFlattener. (This can also be done in the XMLFragmenter, but for learning this might be easier to do step by step.) Match logger.

Edit: added workspace template

parsexml.fmwt

The messages are there, but the standard behavior in FME is to either create an attribute or a list in each "logger" feature depending on the number (cardinality) of "message" objects. If there is only one "message", then the object is output as regular attributes, but if there are several "message" objects per "logger", then a list is output. This is a bit cumbersome because you will have to treat the two cases differently, unless you always have multiple messages per logger.

However, it is possible to tell FME to always use a list for a specific element, here's an example on how to force the "messages" as a list:

In the dialog "XML Flatten options" you will have to toggle advanced mode and type the following to specify the cardinality of the message objects:

cardinality="*/messages/message{}/+ /+"

This means that all "message" objects inside the "messages" element should be rendered as an FME list regardless of the number. You can then use a ListExploder to get all the messages per logger:

The messages are there, but the standard behavior in FME is to either create an attribute or a list in each "logger" feature depending on the number (cardinality) of "message" objects. If there is only one "message", then the object is output as regular attributes, but if there are several "message" objects per "logger", then a list is output. This is a bit cumbersome because you will have to treat the two cases differently, unless you always have multiple messages per logger.

However, it is possible to tell FME to always use a list for a specific element, here's an example on how to force the "messages" as a list:

In the dialog "XML Flatten options" you will have to toggle advanced mode and type the following to specify the cardinality of the message objects:

cardinality="*/messages/message{}/+ /+"

This means that all "message" objects inside the "messages" element should be rendered as an FME list regardless of the number. You can then use a ListExploder to get all the messages per logger:

First of all, there's an invalid character on lines 59456 and 59473, e.g.:

<notes>IPID - 10762190
SS - 42109, 25614

Actuator Max 30m ? Min 15m</notes>

You might want to e.g. use the StringReplacer to fix those first.

You can then read the XML like this (or similar using the FeatureReader):

Result:

The messages are there, but the standard behavior in FME is to either create an attribute or a list in each "logger" feature depending on the number (cardinality) of "message" objects. If there is only one "message", then the object is output as regular attributes, but if there are several "message" objects per "logger", then a list is output. This is a bit cumbersome because you will have to treat the two cases differently, unless you always have multiple messages per logger.

However, it is possible to tell FME to always use a list for a specific element, here's an example on how to force the "messages" as a list:

In the dialog "XML Flatten options" you will have to toggle advanced mode and type the following to specify the cardinality of the message objects:

cardinality="*/messages/message{}/+ /+"

This means that all "message" objects inside the "messages" element should be rendered as an FME list regardless of the number. You can then use a ListExploder to get all the messages per logger:

Hi @david_r,

Looks like I'm going to have to parameterize my api in order to fire a list of values into the url string. I can't get this to work with a feature reader - do I have to use a HTTPCaller and then use the XML fragmenter?