I have many parquet files stored in an s3 bucket; when I use the s3 connector to download the parquet file, I am able to read it successfully, but passing the s3 uri (“s3://”) raises an error:
PARQUET reader: PARQUET reader: Failed to open file 's3:\my-bucket-s3-test\fme-test\test.parquet' for reading. Please ensure that the file exists and you have sufficient privileges to read it
Failed to obtain any schemas from reader 'PARQUET' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information
When I try to pass the url:
PARQUET reader: PARQUET reader: Failed to open file 'HTTPS:\s3.us-west-2.amazonaws.com\my-bucket-s3-test\fme-test\test.parquet' for reading. Please ensure that the file exists and you have sufficient privileges to read it
When I use the feature reader, regardless of whether or not I specify a web connection, it converts forward slashes to back slashes. How do I work with this?
Context:
There will be 1000s of files in this bucket. I am using “list” to generate a feature of the pathnames, then using automations to process the files in this bucket. I do not want to download all of these files, given the memory strain, and I need to use attributes to pass the uris to next part of the process
