Skip to main content
Solved

How can I define that my FME Workbench should process file after file?

ewb_fme
Contributor
Forum|alt.badge.img+4

Here is my use case:

 

I have created a workflow that works well and can handle multiple files. However, when I publish this workflow to FME Flow, I have the following problem when I copy and paste multiple files at once:

 

 

This is a bit annoying and I am looking for another solution. Maybe if I can run the workflow file by file. But how is this possible with FME Flow?

 

Does anyone have a recommendation on how to solve this problem?

*****************

Update, 2023-10-26

How to proceed, if FME Flow should process the files in a defined order? I would like that FME starts with the oldest file first, then the second oldest and so on. E.g. I copy 4 files at the same time in a folder and right now FME just picks the files randomly (not from latest to oldest or the other way round).

Best answer by siennaatsafe

Hi @ewb_fme​ ,

 

I'm not sure how your workspace currently works but you could make a published parameter for the file. Then, feed the filename in from the directory watch. When two files arrive in the directory, the workspace will run twice, 1 per file.

 

2023-10-16_16-42-11 

See this article for details https://community.safe.com/s/article/run-a-workspace-when-data-arrives-in-a-directory

View original
Did this help you find an answer to your question?

5 replies

siennaatsafe
Safer
Forum|alt.badge.img+11
  • siennaatsafeSafer
  • Safer
  • Best Answer
  • October 17, 2023

Hi @ewb_fme​ ,

 

I'm not sure how your workspace currently works but you could make a published parameter for the file. Then, feed the filename in from the directory watch. When two files arrive in the directory, the workspace will run twice, 1 per file.

 

2023-10-16_16-42-11 

See this article for details https://community.safe.com/s/article/run-a-workspace-when-data-arrives-in-a-directory

ewb_fme
Contributor
Forum|alt.badge.img+4
  • ewb_fmeContributor
  • Author
  • Contributor
  • October 17, 2023
siennaatsafe wrote:

Hi @ewb_fme​ ,

 

I'm not sure how your workspace currently works but you could make a published parameter for the file. Then, feed the filename in from the directory watch. When two files arrive in the directory, the workspace will run twice, 1 per file.

 

2023-10-16_16-42-11 

See this article for details https://community.safe.com/s/article/run-a-workspace-when-data-arrives-in-a-directory

@siennaatsafe​ Thank you very much for your input!!! This is exactly what I was looking for! 😀

ewb_fme
Contributor
Forum|alt.badge.img+4
  • ewb_fmeContributor
  • Author
  • Contributor
  • October 26, 2023

@siennaatsafe​ 

I have an additional question regarding this topic. What if I have to precess these files according to their creation date? For example when I copy 4 files to this folder, then I would like that FME Flow would process the oldest file first, then the second oldest and so on. Right now, it processes the files randomly... Do you have an idea how to solve this?

siennaatsafe
Safer
Forum|alt.badge.img+11

Hi @ewb_fme​ 

Unfortunately, there is no easy way to do this with the Directory Watch trigger.

 

However, you could do this, if you wanted to use a schedule.

 

In a workspace, you'll want to use a directory and filepaths reader (to get the filenames and the times submitted), a datetime converter (to convert to epoch time), a sorter, and an Automations writer.2023-10-27_13-26-30This will sort your files by the time they arrive and the Automations writer can be used to pass the filenames into the following workspace through the automation.

 

2023-10-27_13-32-21 

Now, if you use this method, you'll have to add some logic to not reprocess files, this could either be moving them out of the directory after they've been processed or you could have a database that tracks which files have already been processed.

ewb_fme
Contributor
Forum|alt.badge.img+4
  • ewb_fmeContributor
  • Author
  • Contributor
  • October 30, 2023
siennaatsafe wrote:

Hi @ewb_fme​ 

Unfortunately, there is no easy way to do this with the Directory Watch trigger.

 

However, you could do this, if you wanted to use a schedule.

 

In a workspace, you'll want to use a directory and filepaths reader (to get the filenames and the times submitted), a datetime converter (to convert to epoch time), a sorter, and an Automations writer.2023-10-27_13-26-30This will sort your files by the time they arrive and the Automations writer can be used to pass the filenames into the following workspace through the automation.

 

2023-10-27_13-32-21 

Now, if you use this method, you'll have to add some logic to not reprocess files, this could either be moving them out of the directory after they've been processed or you could have a database that tracks which files have already been processed.

@siennaatsafe​ thank you very much for your answer!

Another option I tried was

  1. to use a "Resource or Network Directory" trigger with CREATE and DELETE
  2. do the sorting in Workbench as you suggested
  3. and then instead of using an Automation Writer, just do the data manipulation directly in the same workbench and check the "Skip if Job In Progress" box.
  4. After the workbench is done I move the file and this triggers again point 1 (because of the DELETE option)

Reply


Helpful Members This Week

Community Stats

31,209
Posts
118,337
Replies
38,814
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.