I am trying to download all rasters from a recent USGS LiDAR Project. The files can be found here: The National Map (prd-tnm.s3.amazonaws.com)
Is there way using FME to download all the files in this folder to my local machine?
Solved
Download USGS Tiffs in Bulk
Best answer by redgeographics
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+61- Celebrity
- Best Answer
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
Unfortunately that list is Javascript generated, so the HTTPCaller can't get to it.
You could manually copy and paste the list into a Text file, then read that and construct the url from the filename, then use HTTPCaller to download the files. Unfortunately that's not a very dynamic workflow so if this is something you need to do on a regular basis it's going to be cumbersome.
You could try the S3Connector, but you'd have to know the bucket name and region.
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
+61- Celebrity
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
You can use a HTTPCaller to get each file and save it (the "Save response body to" parameter. If you just need to download that's the one you need. If you want to do something in FME with the data right away you could use the ImageFetcher instead.
You may need to add a Decelerator before the HTTPCaller or ImageFetcher to avoid overloading the server or hitting limits on it.
+64- Celebrity
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
Something i've started doing is setting a random number (5-15s tends to sit well) on each feature and using that as the Decelerator. I've been temporarly bared from sites using a fixed time (10s) before.
+61- Celebrity
Okay thanks. I was able to copy paste the HTML data into a text file and do some REGEX to create a table with the URL and file names.
I am not familiar with the HTTPCaller, can I use this with the GET method then write to Geotiff?
Also saw the ImageFetcher transformer. Will this work?
That's a really smart solution!