I have a flow Import_Points.fmw with two parameters (source and version). It is possible execute simultaneous .bat (point_1.bat, point_2.bat,...) to the flow Import_Points.fmw?
See image please.
Parallel Processing - Database versioned
I need to run a parallel flow to import 8 million points in a database versioned .
Userlevel 2
+17
- Contributor
- 7538 replies
-
5 November 2014
Hi,
Yes, multiple FME processes can be executed simultaneously. But I would use the WorkspaceRunner to run the "Import_Points.fmw" workspace, rather than batch files.
If you set "No" to the "Wait for Job to Complete" parameter, the transformer will launch an FME process immediately when an input feature has arrived. So multiple processes (maximum 7) can be executed for every input features.
You can set the required parameters (source and version) as attributes of input features and pass them to each FME process through the transformer.
If I do that, I would create a table with CSV format (a line contains "source,version"), read the table with the CSV reader, and send the features to the WorkspaceRunner.
Takashi
Userlevel 2
+17
- Contributor
- 7538 replies
-
12 December 2014
No, you don't need to create 8 Import_Points workspaces.
Basic mechanism of the WorkspaceRunner is same as using batch files, but you can set parameters more flexibly.
If the existing "Import_Points.fmw" publishes these 4 parameters,
data_source
alternative
port
service
create a csv table having 4 columns and 8 data rows, like this.
data_source,alternative,port,service
Point_1.ffs,Point_1,30000,fme_1
Point_2.ffs,Point_2,30001,fme_2
...
Point_8.ffs,Point_8,30008,fme_8
Then, create a workspace to read the csv table with the CSV reader and send the features (csv rows) to a WorkspaceRunner; specify the file path of the "Import_Points.fmw" to the "FME Workspace" parameter of the WorkspaceRunner. The published parameter names will appear automatically when you set the "FME Workspace" parameter.
You can then set the published parameters to the csv attribute values.
Save the workspace and run it. The WorkspaceRunner will run "Import_Points.fmw" 8 times repeatedly for each csv row (i.e. for each parameter set).
Note that the maximum number of FME processes that can be concurrently executed is limited to seven. So, in this case, the 8th run has to wait for completion of a previous process.
Hope this helps.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.