Skip to main content

I am trying to convert a Google KML file to ESRI Shapefile and to include the ExtendedData as attributes.

European Space Agency is publishing planned satellite observation scenario as a KML file that has several attributes in the ExtendedData tag (see ). These attributes are always the same, as here:
<ExtendedData>
   <Data name="SatelliteId">
      <value>S1A</value>
   </Data>
   <Data name="DatatakeId">
      <value>1C269</value>
   </Data>
   <Data name="Mode">
      <value>EW</value>
   </Data>
   <Data name="Swath">
      <value>NA</value>
   </Data>
   <Data name="Polarisation">
      <value>DH</value>
   </Data>
   <Data name="ObservationTimeStart">
      <value>2017-06-06T20:19:16</value>
   </Data>
   <Data name="ObservationTimeStop">
      <value>2017-06-06T20:20:32</value>
   </Data>
   <Data name="ObservationDuration">
      <value>76</value>
   </Data>
   <Data name="OrbitAbsolute">
      <value>16918</value>
   </Data>
   <Data name="OrbitRelative">
      <value>46</value>
   </Data>
  </ExtendedData>
What is the easiest way to extract the attributes and to write them to Shapefile? With Inspector I can see the attributes for KML, but I do not know how to write them to Shapefile.

0684Q00000ArENSQA3.jpg

There are multiple ways to do this, but I'd choose a PythonCaller, since this is the cleanest option imho. You would have to expose the attributes somehow, because FME needs to know what it has to write out. I've chosen to do this in the PythonCaller as well, because the schema of your data seems to be fixed anyway.

 

Have a look at my solution in the attachment. You can now simply add a Shapefile writer to it in Automatic mode, but be aware that the attribute names can be too long for a Shapefile and might get truncated.

In case you don't have FME 2017, this is the Python code you can paste in the PythonCaller. Be sure to set the "Class or Function to Process Data" value to "processKmlData", so that it calls the feature function.

import fme
import fmeobjects
 
def processKmlData(feature):
    names = feature.getAttribute('kml_data{}.name')
    values = feature.getAttribute('kml_data{}.value')
    if not (isinstance(names, list) and isinstance(values, list)):
        return
    num_attributes = len(names)
    if num_attributes != len(values):
        return
    for i in range(num_attributes):
        feature.setAttribute(names i], valuesii])

 


There are multiple ways to do this, but I'd choose a PythonCaller, since this is the cleanest option imho. You would have to expose the attributes somehow, because FME needs to know what it has to write out. I've chosen to do this in the PythonCaller as well, because the schema of your data seems to be fixed anyway.

 

Have a look at my solution in the attachment. You can now simply add a Shapefile writer to it in Automatic mode, but be aware that the attribute names can be too long for a Shapefile and might get truncated.

In case you don't have FME 2017, this is the Python code you can paste in the PythonCaller. Be sure to set the "Class or Function to Process Data" value to "processKmlData", so that it calls the feature function.

import fme
import fmeobjects
 
def processKmlData(feature):
    names = feature.getAttribute('kml_data{}.name')
    values = feature.getAttribute('kml_data{}.value')
    if not (isinstance(names, list) and isinstance(values, list)):
        return
    num_attributes = len(names)
    if num_attributes != len(values):
        return
    for i in range(num_attributes):
        feature.setAttribute(names i], valuesii])

 

This is exactly what is needed. I did not know that Python can solve the problem so elegantly.

 


FYI @krostir: I have just uploaded a custom transformer to the FME Hub that achieves the above in a more generic/elegant way. Check out the ListKeyValuePairExtractor!


FYI @krostir: I have just uploaded a custom transformer to the FME Hub that achieves the above in a more generic/elegant way. Check out the ListKeyValuePairExtractor!

I am trying to use the ListKeyValuePairExtractor but could not fugure out how. Do you have an example I could use?

 

 


I am trying to use the ListKeyValuePairExtractor but could not fugure out how. Do you have an example I could use?

 

 

Hi @krostir,

 

 

I am sorry to hear that! I spent a lot of time trying to write clear documentation... Unfortunately, those key and value list parameters are a bit confusing, but that's mainly because of a limitation of the FME GUI types. As far as I know, there's no way to create an input parameter for a single nested list.

 

Anyway, did you see that there is a Download Test button next the Download button on the FME Hub page? In that workspace, I create a couple of list types so you can see what it does.

 

 

For your specific case though, here is another example workspace that uses the ListKeyValuePairExtractor and your KML example file! On my machine, it took a long time to run by the way, but that was caused by some HTTP connection timeouts on the KML Reader.


Hi @krostir,

 

 

I am sorry to hear that! I spent a lot of time trying to write clear documentation... Unfortunately, those key and value list parameters are a bit confusing, but that's mainly because of a limitation of the FME GUI types. As far as I know, there's no way to create an input parameter for a single nested list.

 

Anyway, did you see that there is a Download Test button next the Download button on the FME Hub page? In that workspace, I create a couple of list types so you can see what it does.

 

 

For your specific case though, here is another example workspace that uses the ListKeyValuePairExtractor and your KML example file! On my machine, it took a long time to run by the way, but that was caused by some HTTP connection timeouts on the KML Reader.

Hi @sander_s,

 

 

this is impressive. It works very well. Thank you.

 

 

However, for an unknown reason, I do not see the Download Test button in the FME Hub.

 


Hi @sander_s,

 

 

this is impressive. It works very well. Thank you.

 

 

However, for an unknown reason, I do not see the Download Test button in the FME Hub.

 

Hmmm... it seems that only I, the transformer uploader, can see that... In the past people could download the test workspace.

 

By the way, I just updated the transformer. No version upgrade, so not necessary to download it again. I merely improved the documentation a little.

 


Reply