Question

read .kml files from a url

  • 14 November 2018
  • 11 replies
  • 29 views
G

I am new to fme and trying to read kml files from an http address, where the files may be spread out through different directories.

I would like to recurse through the directories and find all the .kml files and write the placemarks out to a AGOL feature service.

My issue is I do not know how to gather the .kml files from the http address. I keep getting errors about reading from the address and 404 with the http response.

 

Should I be using the HTTPCaller, or can I simply use the FeatureReader or a direct KMLReader?

I have tried all three, but with no success. Any help would be appreciated.

11 replies

redgeographics
Rank
Userlevel 5
Badge +25

It seems wildcards and url's don't mix. What you can do is use a HTMLExtractor to grab the directory listing of http://oer.hpc.msstate.edu/okeanos/, then parse that to get the directory names, throw each one of those into another HTMLExtrator to get those directory listings, parse out the KML filenames and use a FeatureReader to grab those. A bit clunky I guess but it should work.

david_r
Rank
Userlevel 4

HTTP as a protocol has no notion of files and folders, so you cannot use wildcards in URLs unless the server explicitely supports it e.g. as part of an API.

You need to find a way to know the URL (path) to each and every .kml file before using the KML reader or HTTPCaller. How you would go about doing this would depend on the http server you're reading from.

If there is e.g. an html document that lists all the .kml files, you could use the HTMLExtractor or some Python code to get to all the .kml files and download them by iteration.

nielsgerrits
Rank
Userlevel 6
Badge +32

It seems wildcards and url's don't mix. What you can do is use a HTMLExtractor to grab the directory listing of http://oer.hpc.msstate.edu/okeanos/, then parse that to get the directory names, throw each one of those into another HTMLExtrator to get those directory listings, parse out the KML filenames and use a FeatureReader to grab those. A bit clunky I guess but it should work.

Can confirm this works excellent.

G

It seems wildcards and url's don't mix. What you can do is use a HTMLExtractor to grab the directory listing of http://oer.hpc.msstate.edu/okeanos/, then parse that to get the directory names, throw each one of those into another HTMLExtrator to get those directory listings, parse out the KML filenames and use a FeatureReader to grab those. A bit clunky I guess but it should work.

I am trying this, but could you add a simple image of the workflow, I can not seem to get it to work.

nielsgerrits
Rank
Userlevel 6
Badge +32

Here you go :)

getfiles2018.fmw

G

Here you go :)

getfiles2018.fmw

Wow! Thank you so much! Being new to FME I had not gotten too far with this!! Guess this means I have lots of learning to do!!

nielsgerrits
Rank
Userlevel 6
Badge +32

Wow! Thank you so much! Being new to FME I had not gotten too far with this!! Guess this means I have lots of learning to do!!

It is always hard to estimate ones experience level so don't be embarrassed to ask when the solution is too difficult to understand. Let me know if you need more help.

G

It is always hard to estimate ones experience level so don't be embarrassed to ask when the solution is too difficult to understand. Let me know if you need more help.

Hi Niels,

Been a bit since I tested this, and I have upgraded my build version of FME workbench and now this is no longer working?

FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN64

It seems to be hanging up on the HTMLExtractor2 step and rejecting everything that it previously was reading fine?

Can you help me with this?

nielsgerrits
Rank
Userlevel 6
Badge +32

Hi Niels,

Been a bit since I tested this, and I have upgraded my build version of FME workbench and now this is no longer working?

FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN64

It seems to be hanging up on the HTMLExtractor2 step and rejecting everything that it previously was reading fine?

Can you help me with this?

Heya @ginabrewer I just downloaded and installed the same release and could run the getfiles2018.fmw attached above without any issue... But I can check your workspace if you can share it.

G

Heya @ginabrewer I just downloaded and installed the same release and could run the getfiles2018.fmw attached above without any issue... But I can check your workspace if you can share it.

I did not change anything from what you sent.....so not sure why it is rejecting the file at the HTML2Extractor for me? Here is the workspace.

getkmlfiles_okeanos.fmw

nielsgerrits
Rank
Userlevel 6
Badge +32

I did not change anything from what you sent.....so not sure why it is rejecting the file at the HTML2Extractor for me? Here is the workspace.

getkmlfiles_okeanos.fmw

Weird. I can't reproduce the problem. Does the log mention warning or error messages?

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.