I am trying to set up a process that downloads new zip files containing shapefiles from a Google Bucket. The bucket contains four folders at the top level and in each of those folders are folders for individual project areas. In those project folders there are mp4s, pdfs and zip files with the shape files. I would like to just download those zip files, and keep them grouped by project, and none of the others every time the flow is triggered. So far I have figured out out how to list the files and filter out everything else not a zip file. The list is great but I want to download them and not just list them. This is my first time working with Google Cloud Storage buckets.
Now that you've filtered for zips you should be able to feed the path of the zip file in as the path for the Path to Download in a new Connector.
You might want to define an Output folder where all zips will get downloaded too. If you need to keep the folder structure of the online storage then you should be able to combine the path of your output folder with the path on the cloud.
You want to use the download as file option rather than folder in your case.
Use a sampler to test with only a handful of zips before you run the whole thing to make sure it's working as expected
Now that you've filtered for zips you should be able to feed the path of the zip file in as the path for the Path to Download in a new Connector.
You might want to define an Output folder where all zips will get downloaded too. If you need to keep the folder structure of the online storage then you should be able to combine the path of your output folder with the path on the cloud.
You want to use the download as file option rather than folder in your case.
Use a sampler to test with only a handful of zips before you run the whole thing to make sure it's working as expected
How do I feed that path into the second GoogleCloudStorageConnector? This connector has got me stumped. I have fed the sampled list into the connector and then set it to download as a file to a local test folder and I am not getting anything to download in the destination folder. Any suggestions?
Following the connector transformer, which currently has an output called _download_path, place a featureReader
read downloaded Google Cloud Storage data using the FeatureReader, or upload data written by the FeatureWriter to Google Cloud Storage
With the FeatureReader, specify the main format you are downloading from GoogleCloudStorage and the dataset will be _download_path attribute, selecting the down arrow to find the initator attributes
