• EN
Question

Help with set up of processing many datasets with same inputs to outputs into relevant folders

  • 20 October 2023
  • 6 replies
  • 6 views
C

I will be receiving a number of datasets that I need to run through the same process. I would like to figure out how I can set up the workbench to be able to put through several datasets at once time. There will be the same 4 inputs that need to be run through the FME workflow I have set up to output 3 files.

I'm looking at working with something like this:

imageIf anyone has an example I could maybe take a look at to help set this up or have any tips/advice for doing this, it would be much appreciated! Many thanks in advance!

6 replies

nielsgerrits
Rank
Userlevel 6
Badge +34

I recently created a sample for a similar question. Can you check that one and see if this is what you need? You probably need to add a FilenamePartExtractor in the mix to get the root folder.

C

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

nielsgerrits
Rank
Userlevel 6
Badge +34

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

 

It depends... :) hard to say without more info, but almost everything is possible if it has logic in it.

 

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

 

This is also possible. You can use a parent workspace scanning the folders, starting a child workspace using a workspacerunner. In the child workspace, convert your input datasets to published parameters, so you can access those with the workspacerunner.

C

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

 

It depends... :) hard to say without more info, but almost everything is possible if it has logic in it.

 

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

 

This is also possible. You can use a parent workspace scanning the folders, starting a child workspace using a workspacerunner. In the child workspace, convert your input datasets to published parameters, so you can access those with the workspacerunner.

Do you by any chance have an example of parent and child workspace and how it interacts? I find it's easier for me to look at an example to try and replicate for what I need :) Thanks!

nielsgerrits
Rank
Userlevel 6
Badge +34

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

 

It depends... :) hard to say without more info, but almost everything is possible if it has logic in it.

 

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

 

This is also possible. You can use a parent workspace scanning the folders, starting a child workspace using a workspacerunner. In the child workspace, convert your input datasets to published parameters, so you can access those with the workspacerunner.

Sure :)

1 Attachment
C

Thanks @nielsgerrits​! I took a look and this is useful. One difference is that, I would need the Inputs (1-4) to feed into specific points within the workbench so may not work with this example if it scans folder input for specific file types. If a FME workbench is set up already which takes 1 input at a time, would it be relatively easy to convert it to enable multiple?

 

It depends... :) hard to say without more info, but almost everything is possible if it has logic in it.

 

Thinking about this, maybe it's easier for me to create an additional workbench that allows me to read the first set of inputs, and output then move onto the next folder etc rather than all at once

 

This is also possible. You can use a parent workspace scanning the folders, starting a child workspace using a workspacerunner. In the child workspace, convert your input datasets to published parameters, so you can access those with the workspacerunner.

Thank you for your help!

Reply


Community Stats

28,866
Posts
109,981
Replies
36,670
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.