read .kml files from a url

It seems wildcards and url's don't mix. What you can do is use a HTMLExtractor to grab the directory listing of http://oer.hpc.msstate.edu/okeanos/, then parse that to get the directory names, throw each one of those into another HTMLExtrator to get those directory listings, parse out the KML filenames and use a FeatureReader to grab those. A bit clunky I guess but it should work.

HTTP as a protocol has no notion of files and folders, so you cannot use wildcards in URLs unless the server explicitely supports it e.g. as part of an API.

You need to find a way to know the URL (path) to each and every .kml file before using the KML reader or HTTPCaller. How you would go about doing this would depend on the http server you're reading from.

If there is e.g. an html document that lists all the .kml files, you could use the HTMLExtractor or some Python code to get to all the .kml files and download them by iteration.

Can confirm this works excellent.

I am trying this, but could you add a simple image of the workflow, I can not seem to get it to work.

Here you go :)

getfiles2018.fmw

Wow! Thank you so much! Being new to FME I had not gotten too far with this!! Guess this means I have lots of learning to do!!

It is always hard to estimate ones experience level so don't be embarrassed to ask when the solution is too difficult to understand. Let me know if you need more help.

Hi Niels,

Been a bit since I tested this, and I have upgraded my build version of FME workbench and now this is no longer working?

FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN64

It seems to be hanging up on the HTMLExtractor2 step and rejecting everything that it previously was reading fine?

Can you help me with this?

Heya @ginabrewer I just downloaded and installed the same release and could run the getfiles2018.fmw attached above without any issue... But I can check your workspace if you can share it.

I did not change anything from what you sent.....so not sure why it is rejecting the file at the HTML2Extractor for me? Here is the workspace.

getkmlfiles_okeanos.fmw

Weird. I can't reproduce the problem. Does the log mention warning or error messages?