modify data download destination directory (Amazon S3 bucket)

Thanks Ken,

 

That was really helpful and will definitely work for us, but...the worker workspace writes a raster output and a text file output.  I want to include both of these outputs in a zip file then upload the zip file to the S3 bucket.  How do I create that zip file with there doesn't seem to be a FME_SERVER_DEST_DIR parameter for the worker workspace.

 

I've been playing with a python shutdown script on the worker workspace to try and do the zipping (https://knowledge.safe.com/articles/How_To/Zip-your-Output-with-FME) but don't know where to point it.

 

These are examples of the output locations for the text file and grid:

 

/mnt/fme_temp/fmeengines/localhost_Engine4/FME_NEDF_DEMClipZipShip_Worker_1441339087140_20082/GENERIC_1/NEDF_test

 

/mnt/fme_temp/fmeengines/localhost_Engine4/FME_NEDF_DEMClipZipShip_Worker_1441339087140_20082/TEXTLINE_1/readme_metadata.txt

 

Ideally I'd like to zip the files under this directory

 

/mnt/fme_temp/fmeengines/localhost_Engine4/FME_NEDF_DEMClipZipShip_Worker_1441339087140_20082

 

but there doesn't seem to be a parameter defining this location.

 

 

I've noticed the ZipArchiver transformer but have an older build so don't have access to it yet.

 

 

Any ideas?

 

 

Thanks

 

Rob

 

 

Hi Rob,

 

Here is a suggestion I just tried. By the way can you please tell me what version of FME Desktop and Server you are working with?

 

 

All file and directory writers in FME have the ability to set a zip file as ouput. You can set multiple writers to the same zip file name and all of the output will go into the one zip file. (To create a more complex zip file structure you'll need to use Dataset Fanout but we can come back to that.)

 

You can write this zip file to the FME Server Resources temp directory using an existing FME Server system parameter which points to it (the true path is hidden). And then the S3Uploader can get it from there. To answer your questions the FME_SERVER_DEST_DIR parameter is only created if using the actual Data Download Service.

 

 

Here are the basic steps:

 

 

1. In the Worker workspace any writers should have their destination data set parameter set like this:

 

$(FME_SHAREDRESOURCE_TEMP)\\output.zip

 

This way all output goes to a single file in your shared resources temp directory.

 

 

2. Then you will have to change a couple of small things in the master The setting in the FMEServerJobSubmitter which says "Output Data Location" needs to be changed to "As Directed in the Workspace".  Now you need to change the path to the file for the S3 uploader. Do this by creating a FilePath attribute with an AttributeCreator and set it to $(FME_SHAREDRESOURCE_TEMP)/output.zip

 

In the S3Uploader change the Source File name to this new FilePath attribute and change the key to output.zip

 

 

You'll likey need to create a hashed zip file name so multiple users aren't writing to the same zip file and the same time but we can come back to that if you think this approach will work for you.

 

Hope this helps

 

Ken

Thanks Ken,

 

That worked perfectly.  Great little tip about the ability of FME writers to generate zip file...didn't know that!

 

For now used an attribute creater in the master workspace to create a unique output path to the zip file $(FME_SHAREDRESOURCE_TEMP)DataExtract$(FME_JOB_ID).zip

 

and pass its value to the worker workspace via a published paramter.  I've added the job ID to the file name to make it unique.

 

Thanks again for all your help Ken,

 

Rob