I'd like to run this quarterly using parameters for the date specific strings.
Thanks!
Page 1 / 1
Hi @bobw
Configure the S3Connector like this if you want to download a file from the ookla-open-data bucket.
Since you want to use parameters for the date components in the path, use a year and quarter user parameter and create the path as an attribute value. Use a conditional to map the month value based on quarter (eg. 1=01, 2=04, 3=07, etc) if the zip files are named consistently. The path attribute would look similar to this:
shapefiles/performance/type=fixed/year=$(Year)/quarter=$(Quarter)/$(Year)-@Value(month)-01_performance_fixed_tiles.zip
An alternative would be to use two consecutive S3Connectors if the file names are inconsistent. The first to list any files contained within a folder year/quarter and then use another to actually download the file based on the path from the first S3Connector.
Hi @bobw
Configure the S3Connector like this if you want to download a file from the ookla-open-data bucket.
Since you want to use parameters for the date components in the path, use a year and quarter user parameter and create the path as an attribute value. Use a conditional to map the month value based on quarter (eg. 1=01, 2=04, 3=07, etc) if the zip files are named consistently. The path attribute would look similar to this:
shapefiles/performance/type=fixed/year=$(Year)/quarter=$(Quarter)/$(Year)-@Value(month)-01_performance_fixed_tiles.zip
An alternative would be to use two consecutive S3Connectors if the file names are inconsistent. The first to list any files contained within a folder year/quarter and then use another to actually download the file based on the path from the first S3Connector.
Works like a charm. I like how the S3Connector drills into the Path for the bucket. Very cool. Thank you.