Question

How do you avoid overwriting a common output zip file when used by multiple Job Submitter jobs?

  • 12 August 2016
  • 3 replies
  • 1 view
R
Badge

Hi all

I'm trying to create a single output zip file that is written to by one or more Job Submitter jobs. I've got a service that allows the user to select multiple feature types from a POSTGIS db from which they want to extract data within selected bounding coordinates. The list of selected feature types is split prior to entering an FMEServerJobSubmitter transformer that kicks off a workspace that does the actual data extract. The first workspace passes the output zip location to the data extraction workspace so that they can write to a common destination.

In testing, the zip is created and contains data but only from the last feature type to be processed. Looks like each time the data extraction workspace runs, it over-writes the zip file created by the data extraction job that ran before it.

I've tested with the Wait for Server Job to Complete parameter on the FMEServerJobSubmitter set to "Yes" and "No" and it doesn't make a difference.

I'm using the Job Submitter service because the data is pushed to AWS S3 prior to sending a download notification to the user.

Any suggestions?

Thanks

Rob

3 replies

redgeographics
Rank
Userlevel 5
Badge +25

I don't think that's possible actually. Can you have the individual child processes write to normal (non-zipped) files then zip them up afterwards (there's a Python library for that) before pushing them to S3?

mark2atsafe
Rank
Userlevel 4
Badge +25

Is there a specific need to split up the processing? Why not have a single workspace read all the feature types and write them out? In fact, why not one workspace to read the data and push it to S3? The FeatureWriter and S3Uploader transformers would do this easily enough. Is it a performance issue?

R
Badge

I was thinking that splitting the process would allow me to report more easily the reason why a feature type job failed or returned nothing. We're sending an email to the user with a data download link and reporting the results of the process in the email. I was thinking that it'd be useful information for the user and for ourselves (we're also capturing job info in Dynamo db).

I'll look into using the FeatureWriter. Maybe I can simplify it into one workspace.

Reply


Community Stats

28,716
Posts
109,494
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.