Skip to main content

I am trying to download all rasters from a recent USGS LiDAR Project. The files can be found here: The National Map (prd-tnm.s3.amazonaws.com)

 

Is there way using FME to download all the files in this folder to my local machine?

Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.

You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.

You could try the S3Connector, but you'd have to know the bucket name and region.


Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.

You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.

You could try the S3Connector, but you'd have to know the bucket name and region.

TableFMEOkay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.

 

I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?

Also saw the ImageFetcher transformer. Will this work?

 


TableFMEOkay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.

 

I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?

Also saw the ImageFetcher transformer. Will this work?

 

You can use a HTTPCaller to get each file and save it (the "Save response body to" parameter. If you just need to download that's the one you need. If you want to do something in FME with the data right away you could use the ImageFetcher instead.

You may need to add a Decelerator before the HTTPCaller or ImageFetcher to avoid overloading the server or hitting limits on it.


TableFMEOkay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.

 

I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?

Also saw the ImageFetcher transformer. Will this work?

 

Something i've started doing is setting a random number (5-15s tends to sit well) on each feature and using that as the Decelerator. I've been temporarly bared from sites using a fixed time (10s) before.


TableFMEOkay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.

 

I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?

Also saw the ImageFetcher transformer. Will this work?

 

That's a really smart solution!


Reply