S3 connector not uploading the empty folders

​@nasir 

Try using these settings. I think you are going to have to load something so that it creates the folder structure. With the settings below it will essentially create a folder that has a file inside of it called “/”. The file could be called anything but it will more or less be a blank file. 

Data Source settings

Upload options

I can confirm this behaviour as of FME 2024.2

It looks like the tool will only upload files which exist. The “Contents Only” check box refers is asking if you want to include the selected folder itself in the upload or just whats inside it.

The (ugly) workaround here would be to create an empty file in each of the empty folders - Then delete the empty files. 

As far as I can tell there is no way to create an empty folder on s3 with FME with the s3Connector. 

thanks for your reply, yeah It would not be good to created empty files and deleted after uploaded.

This isn’t an FME issue, more of a how S3 buckets (object storage) work.

In Windows (NTFS, FAT etc) the file system is truly hierarchical, a folder exists as its own entity. 

S3 Buckets are object based storage, it’s a flat ‘bucket’ of data. It is a key-value store where keys can look like paths with folders. The ‘folder’ structure is really just a display method to make it easier for us (humans) to read.

Windows:

• C:
  • Temp
    • F1
      • file.txt
      • file2.txt
    • F2
    • F3
      • F4
        • file3.txt

S3:

• Bucket
  • Temp/F1/file.txt
  • Temp/F1/file2.txt
  • Temp/F3/F4/file3.txt

Temp/f2 can’t exist as there is no object stored with this prefix

You can definitely have an empty folder in s3, although ​@hkingsbury is totally right, it’s not really a real folder.  

The api call to upload a file looks a bit different to creating an empty folder. I still think it would be worthwhile. It’s pretty important to be able to create empty folders - especially if you are creating a folder structure where you expect other users to have access and upload to.

In the past I remember using the s3 subscription to upload a folder - I’d messed up the workspace and the data weren’t being output correctly some of the output folders were empty. I was pretty confused for a while as to why the the subscription was only uploading part of my output. I spent ages debugging and looking into the s3 upload process only to realize my data was never getting created and empty folders were not getting uploaded.

Thanks, Crystalatsafe for your update, looking forward for this enhancement, as we are designing a flow to upload the obsolete petrel projects to S3 bucket, and those projects have empty folders/subfolders in them, and we want to keep the project structure same as on active storage.