Skip to main content
Question

How do you avoid overwriting a common output zip file when used by multiple Job Submitter jobs?

  • August 12, 2016
  • 3 replies
  • 47 views
R
Forum|alt.badge.img

Hi all

I'm trying to create a single output zip file that is written to by one or more Job Submitter jobs. I've got a service that allows the user to select multiple feature types from a POSTGIS db from which they want to extract data within selected bounding coordinates. The list of selected feature types is split prior to entering an FMEServerJobSubmitter transformer that kicks off a workspace that does the actual data extract. The first workspace passes the output zip location to the data extraction workspace so that they can write to a common destination.

In testing, the zip is created and contains data but only from the last feature type to be processed. Looks like each time the data extraction workspace runs, it over-writes the zip file created by the data extraction job that ran before it.

I've tested with the Wait for Server Job to Complete parameter on the FMEServerJobSubmitter set to "Yes" and "No" and it doesn't make a difference.

I'm using the Job Submitter service because the data is pushed to AWS S3 prior to sending a download notification to the user.

Any suggestions?

Thanks

Rob

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

redgeographics
Celebrity
Forum|alt.badge.img+62

I don't think that's possible actually. Can you have the individual child processes write to normal (non-zipped) files then zip them up afterwards (there's a Python library for that) before pushing them to S3?

FME rocks! \m/
mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • August 12, 2016

Is there a specific need to split up the processing? Why not have a single workspace read all the feature types and write them out? In fact, why not one workspace to read the data and push it to S3? The FeatureWriter and S3Uploader transformers would do this easily enough. Is it a performance issue?

FME Evangelist to the Rich and Famous!!
R
Forum|alt.badge.img
  • rkay
  • Author
  • August 13, 2016

I was thinking that splitting the process would allow me to report more easily the reason why a feature type job failed or returned nothing. We're sending an email to the user with a data download link and reporting the results of the process in the email. I was thinking that it'd be useful information for the user and for ourselves (we're also capturing job info in Dynamo db).

I'll look into using the FeatureWriter. Maybe I can simplify it into one workspace.

Community Stats

32,208
Posts
122,306
Replies
39,892
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.