Question

How Do I Set the Order of Operations?

  • 13 October 2021
  • 3 replies
  • 35 views
Z
Badge

I need the operations to work in the following order:

  • Terminator
  • DatabaseDeleter
  • Data Load (rest of workspace)

 

I tried using a WorkspaceRunner transformer, having split out the three functions. It works fine if I'm in FME, but if I try to run it using a batch file, it errors out.

 

Ultimately, I want to be able to schedule this through Windows Task Scheduler to run nightly. It will compare the last updated date between two datasets, and terminate if the destination dataset is already up-to-date. Otherwise, it will proceed to delete a subset of the destination dataset (a "partial truncate" for lack of a better term), and then re-load the new data.Workbench

3 replies

hkingsbury
Rank
Userlevel 5
Badge +29

A better approach would be to make use of featurereaders and featurewriters. So the workflow (in one 'branch') would be something like:

  1. Checkdates
    1. If the match no update (i wouldnt then pass it to a terminator as thats going to 'fail' your workbench, leave it empty and have the wb finish successfuly)
    2. if the dates dont match, proceed
  2. Using a featurewriter delete the required rows
  3. once the deletes are complete, use a featurereader to read the data and pass through the rest of the logic

The advantage of using feautrereaders and writers is that you can conditionally control exaclty when data is read (and written/deleted)

In your current workflow, all the readers essentially run reading in all you data irrespective of if you need it or not

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd
Z
Badge

A better approach would be to make use of featurereaders and featurewriters. So the workflow (in one 'branch') would be something like:

  1. Checkdates
    1. If the match no update (i wouldnt then pass it to a terminator as thats going to 'fail' your workbench, leave it empty and have the wb finish successfuly)
    2. if the dates dont match, proceed
  2. Using a featurewriter delete the required rows
  3. once the deletes are complete, use a featurereader to read the data and pass through the rest of the logic

The advantage of using feautrereaders and writers is that you can conditionally control exaclty when data is read (and written/deleted)

In your current workflow, all the readers essentially run reading in all you data irrespective of if you need it or not

Is it going to be something like this? Or is it going to start deleting records from EASTERN_STATES regardless of if the Tester passes or fails? I need to delete only if the Tester passes.

 

Screen Shot 2021-10-14 at 11.02.53 AM

hkingsbury
Rank
Userlevel 5
Badge +29

Is it going to be something like this? Or is it going to start deleting records from EASTERN_STATES regardless of if the Tester passes or fails? I need to delete only if the Tester passes.

 

Screen Shot 2021-10-14 at 11.02.53 AM

Your tester output should be either:

  1. going into a featurereader to read in the EASTERN_STATES data. Then the feature that are read go into a featurewriter to do the delete.
  2. tester output goes straight into the featurewriter and triggers a delete using WHERE 1=1 on the EASTERN_STATES feature class

 

#1 would be more appropriate if you needed to select exactly which features to delete, but will be slower as you need to read all features then delete each one one-by-one

#2 would be faster as you don't need to read in each feature you want to delete and you're pass the load off to the database. However it doesnt make it more difficult to specify which features need deleting

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd

Reply


Community Stats

28,714
Posts
109,490
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.