I am trying to download all rasters from a recent USGS LiDAR Project. The files can be found here: The National Map (prd-tnm.s3.amazonaws.com)
Is there way using FME to download all the files in this folder to my local machine?
Solved
Download USGS Tiffs in Bulk
Best answer by redgeographics
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
+49- Celebrity
- Best Answer
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
redgeographics wrote:
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
+49- Celebrity
tigerwoulds wrote:
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
You can use a HTTPCaller to get each file and save it (the "Save response body to" parameter. If you just need to download that's the one you need. If you want to do something in FME with the data right away you could use the ImageFetcher instead.
You may need to add a Decelerator before the HTTPCaller or ImageFetcher to avoid overloading the server or hitting limits on it.
+51- Celebrity
tigerwoulds wrote:
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
Something i've started doing is setting a random number (5-15s tends to sit well) on each feature and using that as the Decelerator. I've been temporarly bared from sites using a fixed time (10s) before.
+49- Celebrity
tigerwoulds wrote:
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
That's a really smart solution!
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.