Solved

How to create simple S3 folder structure?

  • 8 July 2022
  • 1 reply
  • 31 views

Badge +1

I want to simply add a folder with 3 subfolders to an S3 bucket. I'm using the S3Connector transformer and it seems you need a file to Upload to an S3 folder to create a folder, whereas, it seems to me that you can't only add a folder w/o having a file to upload it. I tried adding a csv file to each subfolder to setup my folder structure in S3 and that works to get the main folder w/ 3 subfolders each containing an empty csv file. I then added 3 more S3Connector transformers w/ the Delete option to remove the csv files from each subfolder thinking then I would have the desired folder setup without any files in them. This doesn't work because when you delete the only csv file from the subfolder, it also removes subfolder leaving nothing left in S3.

 

Is there a way to create the desired s3 folder structure without adding a file? Or is it possible to delete the file from the subfolder without removing the subfolder?

icon

Best answer by debbiatsafe 16 July 2022, 01:45

View original

1 reply

Userlevel 2
Badge +17

Sharing the answer publicly here in case it helps other users running into the same issue.

 

The S3Connector does not currently support creating a folder. Note that this is by-design as S3 has a flat structure instead of a hierarchy as normally seen in a file system (mentioned here in the S3 documentation). S3 supports the folder concept as a way of grouping objects but does not create folders in the usual sense.

 

One possible workaround is to use the PythonCaller and the boto3 module that is already included with an FME install. Alternatively, it may also be possible using the AWS CLI and a SystemCaller.

Reply