Question

Configure Download Zip Filename with User Parameter Value etc.

takashi
Rank
Userlevel 2
Badge +17

FME Server 2016.1

For an on-demand data download service application with FME Server, I'm looking for a way to configure a download zip file name using a published parameter (requestor can set the value) and the Job ID etc.

A possible way I found so far is:

  1. Publish the destination dataset parameter of the main workspace as a published user parameter.
  2. Create another workspace (say 'submitter') containing an FMEServerJobSubmitter which call the main workspace by passing the required destination zip file path.
  3. Run the 'submitter' as a Job Submitter Service.

The destination zip file path can be created using the <results-dirpath>, FME_JOB_ID parameter, and a published user parameter of the 'submitter'. The <results-dirpath> is defined in the "fmeEngineConfig.txt" as the "FME_WORKING_DIR" directive.

It works as expected, but the <results-dirpath> has to be written in the 'submitter' as a full path, and it's a little annoying :(

My questions are:

  1. Is there a way to retrieve the value of the "FME_WORKING_DIR" directive at run-time, like a system parameter? I thought that the "FME_SERVER_DEST_DIR" could be used to retrieve <results-dirpath>, but it will not be defined when the workspace run as a Job Submitter Service.
  2. Is there a smarter way to configure a download zip file name at run time, rather than the name generated by FME Server automatically? It would be ideal if it could be realized with a single workspace that will be run as a Data Download Service.

Regards.

P.S. The goal is to allow users to define any prefix or suffix of download zip file name, so that they can distinguish multiple download files easily. e.g. an expected format is:

<user-defined prefix>_<timestamp>_<job ID>.zip

Any thoughts?

4 replies

takashi
Rank
Userlevel 2
Badge +17

Well, I found a possible way to retrieve the <results-dirpath> in the "submitter" workspace at run-time.

That is, to add a 'dummy' writer to the workspace, which will not write any feature but can be set to the "Include Writers in Download" property when uploading the workspace to FME Server as a Data Download Service.

When a workspace is run as a Data Download Service, the "FME_SERVER_DEST_DIR" parameter will be defined, and then the <results-dirpath> can be retrieved from its value.

L
Badge

Hello takashi,

If having the FME_SERVER_DEST_DIR set for the Job Submitter is a solution for you, you can add a config line in the JOB_SUBMITTER_SERVICE sub section of the fmeEngineconfig.txt file to define your own macro with a generated output folder and then add any prefix or suffix.

Before:

SUB_SECTION JOB_SUBMITTER_SERVICE                                                           \
FME_TRANSFORMATION_LOG_DIR "!FME_SERVER_ROOT!/Logs/engine/current/jobs"                                    \
FME_TRANSFORMATION_LOG_NAME "!FME_AUTO_FILE_NAME_JOBID!.log"              					\
SUCCESS_RESPONSE 0:Translation Successful|NotificationLocation=!FME_AUTO_DIR_NAME!_nw|NumFeaturesOutput=!FME_NUM_FEATURES_OUTPUT!|LogFileName=!FME_TRANSFORMATION_LOG_NAME! \
FAILURE_RESPONSE !FME_ERROR_NUMBER!:!FME_ERROR_MSG!|LogFileName=!FME_TRANSFORMATION_LOG_NAME!


After:

SUB_SECTION JOB_SUBMITTER_SERVICE                                                           \
FME_TRANSFORMATION_LOG_DIR "!FME_SERVER_ROOT!/Logs/engine/current/jobs"                                    \
MACRO_DEF MY_OUTPUT_DIR "!FME_AUTO_DIR_NAME!"                                        \
FME_TRANSFORMATION_LOG_NAME "!FME_AUTO_FILE_NAME_JOBID!.log"              					\
SUCCESS_RESPONSE 0:Translation Successful|NotificationLocation=!FME_AUTO_DIR_NAME!_nw|NumFeaturesOutput=!FME_NUM_FEATURES_OUTPUT!|LogFileName=!FME_TRANSFORMATION_LOG_NAME! \
FAILURE_RESPONSE !FME_ERROR_NUMBER!:!FME_ERROR_MSG!|LogFileName=!FME_TRANSFORMATION_LOG_NAME!


Regards,

Larry

takashi
Rank
Userlevel 2
Badge +17

An outcome from trial-and-error in the last weekend. This may be a possible way to control the destination zip filename in a workspace for the Data Download Service.

1. Create a published parameter to accept a user-defined prefix or suffix for the destination zip filename. e.g. [SUFFIX].

2. Define these three private parameters:

[DEST_DIR_PATH] Scripted Python: Retrieve the destination directory path from the FME_SERVER_DEST_DIR parameter.

import os
try: # FME Server
    dir = os.path.split(FME_MacroValues['FME_SERVER_DEST_DIR'])[0]
except: # FME Desktop
    dir = 'results'
return dir

[DEST_ZIP_FILENAME] Scripted Python: Create the destination zip filename consisting of timestamp, Job ID, and the user-defined suffix.

import datetime
timestamp = datetime.datetime.now().strftime('%y%m%d_%H%M%S')
try: # FME Server
    jobid = FME_MacroValues['FME_JOB_ID']
except: # FME Desktop
    jobid = '0'
return '%s_%s_%s.zip' % (timestamp, jobid, FME_MacroValues['SUFFIX'])

[DEST_DATASET] Text: Return the full path of the destination zip file.

$(DEST_DIR_PATH)/$(DEST_ZIP_FILENAME)

3. Link the Dataset parameter of the writer(s) to the [DEST_DATASET].

4. Add a dummy writer which won't write any feature, so that the workspace can be registered as a Data Download Service without setting the actual writer(s) to the "Include Writers in Download" property.

5. Upload the workspace and register it as a Data Download Service.

0684Q00000ArLNkQAN.png

OK. In the workspace I can overwrite the Email notification for translation SUCCESS using a Text File writer, but have not found a way to overwrite the notification for translation FAILURE.

Also I don't know yet how I can overwrite the response including the download URL, when the workspace has been launched synchronously via REST / JavaScript API.

takashi
Rank
Userlevel 2
Badge +17

Hello takashi,

If having the FME_SERVER_DEST_DIR set for the Job Submitter is a solution for you, you can add a config line in the JOB_SUBMITTER_SERVICE sub section of the fmeEngineconfig.txt file to define your own macro with a generated output folder and then add any prefix or suffix.

Before:

SUB_SECTION JOB_SUBMITTER_SERVICE                                                           \
FME_TRANSFORMATION_LOG_DIR "!FME_SERVER_ROOT!/Logs/engine/current/jobs"                                    \
FME_TRANSFORMATION_LOG_NAME "!FME_AUTO_FILE_NAME_JOBID!.log"              					\
SUCCESS_RESPONSE 0:Translation Successful|NotificationLocation=!FME_AUTO_DIR_NAME!_nw|NumFeaturesOutput=!FME_NUM_FEATURES_OUTPUT!|LogFileName=!FME_TRANSFORMATION_LOG_NAME! \
FAILURE_RESPONSE !FME_ERROR_NUMBER!:!FME_ERROR_MSG!|LogFileName=!FME_TRANSFORMATION_LOG_NAME!


After:

SUB_SECTION JOB_SUBMITTER_SERVICE                                                           \
FME_TRANSFORMATION_LOG_DIR "!FME_SERVER_ROOT!/Logs/engine/current/jobs"                                    \
MACRO_DEF MY_OUTPUT_DIR "!FME_AUTO_DIR_NAME!"                                        \
FME_TRANSFORMATION_LOG_NAME "!FME_AUTO_FILE_NAME_JOBID!.log"              					\
SUCCESS_RESPONSE 0:Translation Successful|NotificationLocation=!FME_AUTO_DIR_NAME!_nw|NumFeaturesOutput=!FME_NUM_FEATURES_OUTPUT!|LogFileName=!FME_TRANSFORMATION_LOG_NAME! \
FAILURE_RESPONSE !FME_ERROR_NUMBER!:!FME_ERROR_MSG!|LogFileName=!FME_TRANSFORMATION_LOG_NAME!


Regards,

Larry

Hi Larry,
Thanks for your answer.

I inserted this macro definition into the JOB_SUBMITTER_SERVICE sub section in the fmeEngineConfig.txt.

MACRO_DEF FME_SERVER_DEST_DIR "!FME_AUTO_DIR_NAME!"    \ 

After rebooting the server, yes, I was able to see that it surely works to pass the 'FME_SERVER_DEST_DIR' to the workspace working as a Job Submitter Service.

This could definitely resolve most part of my question.

Thanks again!

Takashi

Reply


Community Stats

28,663
Posts
109,332
Replies
36,532
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.