I'd like to run this quarterly using parameters for the date specific strings.
Thanks!
Solved
How would I enter something like this in the S3Connector: aws s3 cp --no-sign-request s3://ookla-open-data/shapefiles/performance/type=fixed/year=2022/quarter=1/2022-01-01_performance_fixed_tiles.zip C:\\Users\\me\\GIS\\Data\\Misc\\Ookla\\2022Q1fixed
+5- Contributor
- 19 replies
Best answer by debbiatsafe
Hi @bobw
Configure the S3Connector like this if you want to download a file from the ookla-open-data bucket.
Since you want to use parameters for the date components in the path, use a year and quarter user parameter and create the path as an attribute value. Use a conditional to map the month value based on quarter (eg. 1=01, 2=04, 3=07, etc) if the zip files are named consistently. The path attribute would look similar to this:
shapefiles/performance/type=fixed/year=$(Year)/quarter=$(Quarter)/$(Year)-@Value(month)-01_performance_fixed_tiles.zip
An alternative would be to use two consecutive S3Connectors if the file names are inconsistent. The first to list any files contained within a folder year/quarter and then use another to actually download the file based on the path from the first S3Connector.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+21- Safer
- 648 replies
- Best Answer
Hi @bobw
Configure the S3Connector like this if you want to download a file from the ookla-open-data bucket.
Since you want to use parameters for the date components in the path, use a year and quarter user parameter and create the path as an attribute value. Use a conditional to map the month value based on quarter (eg. 1=01, 2=04, 3=07, etc) if the zip files are named consistently. The path attribute would look similar to this:
shapefiles/performance/type=fixed/year=$(Year)/quarter=$(Quarter)/$(Year)-@Value(month)-01_performance_fixed_tiles.zip
An alternative would be to use two consecutive S3Connectors if the file names are inconsistent. The first to list any files contained within a folder year/quarter and then use another to actually download the file based on the path from the first S3Connector.
+5- Contributor
- 19 replies
Hi @bobw
Configure the S3Connector like this if you want to download a file from the ookla-open-data bucket.
Since you want to use parameters for the date components in the path, use a year and quarter user parameter and create the path as an attribute value. Use a conditional to map the month value based on quarter (eg. 1=01, 2=04, 3=07, etc) if the zip files are named consistently. The path attribute would look similar to this:
shapefiles/performance/type=fixed/year=$(Year)/quarter=$(Quarter)/$(Year)-@Value(month)-01_performance_fixed_tiles.zip
An alternative would be to use two consecutive S3Connectors if the file names are inconsistent. The first to list any files contained within a folder year/quarter and then use another to actually download the file based on the path from the first S3Connector.
Works like a charm. I like how the S3Connector drills into the Path for the bucket. Very cool. Thank you.