Solved

Workspace Callers work in workbench but not from batch

M

Hello,

I've setup four Workbenches, which I successfully get to run in sequence from a "mother" Workbench, which uses a sequence of Workspace Runners, waiting for each one to complete before calling the next one.

It works seamlessly when ran from FME Workbench. But, I really need to automate this (this is what it's all about - I can't always be around to press the "Run" button unfortunately...).

This is my first attempt in calling Workbench(es) via command line, and I just try the obvious:

 

fme "C:\Users\User.Name\Path\Projects Shared\Project_Name\configs\4_Workbench.fmw"

 

I tried first with the "mother" Workbench, and it failed. So then I tried with individual work benches, and they fail too. 

I logged separate log files for each of the five Workbenches (mother + 4  called by the Workspace Runners), and the logs for the 4 Workbenches called have a similar error (I did change some names on purpose, to preserve names of people etc. in paths and file names):

 

ERROR |FeatureReader_4(QueryFactory): The clause 'TREAT_READER_PARAM_AMPERSANDS_AS_LITERALS YES' within 'FACTORY_DEF {*} QueryFactory FACTORY_NAME FeatureReader_4 INPUT FEATURE_TYPE SystemCaller_OUTPUT FCTQUERY_INTERACTION NONE COMBINE_ATTRIBUTES RESULT_ONLY QUERYFCT_ATTRIBUTE_PREFIX <Unused> COMBINE_GEOMETRY RESULT_ONLY ENABLE_CACHE NO QUERYFCT_TABLE_SEPARATOR SPACE READER_TYPE XLSXR READER_DATASET C:<backslash>Users<backslash>USER.NAME<backslash>OneDrive<space>-<space>FOLDER<backslash>Projects<space>Shared<backslash>PROJECT<backslash>data<backslash>SHEET.xlsx QUERYFCT_IDS  READER_DIRECTIVES META_MACROS,SourceCONFIGURATION_DATASET<comma><comma>SourceEXCEL_COL_NAMES<comma>YES<comma>SourceFORCE_DATETIME<comma>NO<comma>SourceSCHEMA<comma><comma>SourceMANYFILE<comma><comma>SourceUSE_SCHEMA_OTHER_FILES<comma>No<comma>SourceADVANCED<comma><comma>SourceSCAN_SCHEMA_IF_DYNAMIC<comma>No<comma>SourceREAD_BLANK_AS<comma>Missing<comma>SourceEXPAND_MERGED_CELLS<comma>Yes<comma>SourceEXPOSE_ATTRS_GROUP<comma><comma>SourceXLSXR_EXPOSE_FORMAT_ATTRS<comma><comma>SourceQUERY_FEATURE_TYPES_FOR_MERGE_FILTERS<comma>Yes<comma>SourceCASE_SENSITIVE_FEATURE_TYPES<comma>YES<comma>SourceALLOW_DOLLAR_SIGNS<comma>YES<comma>SourceCREATE_FEATURE_TABLES_FROM_DATA<comma>Yes<comma>SourceSTRIP_SHEETNAME_SPACES<comma>YES<comma>SourceKEEP_SCHEMA_SETTINGS<comma>YES,METAFILE,XLSXR QUERYFCT_OUTPUT BASED_ON_CONNECTIONS CONTINUE_ON_READER_ERROR YES QUERYFCT_RESULT_TAGS MyField MyField<solidus>Table1 Sheet1 QUERYFCT_SET_FME_FEATURE_TYPE YES READER_PARAMS_WWJD EXCEL_COL_NAMES,YES,CREATE_FEATURE_TABLES_FROM_DATA,Yes,ALLOW_DOLLAR_SIGNS,YES,STRIP_SHEETNAME_SPACES,YES,CASE_SENSITIVE_FEATURE_TYPES,YES,USE_SCHEMA_OTHER_FILES,No,EXPAND_MERGED_CELLS,Yes,READ_BLANK_AS,Missing,KEEP_SCHEMA_SETTINGS,YES,SCAN_SCHEMA_IF_DYNAMIC,No,QUERY_FEATURE_TYPES_FOR_MERGE_FILTERS,Yes,FORCE_DATETIME,NO TREAT_READER_PARAM_AMPERSANDS_AS_LITERALS YES OUTPUT READER_ERROR FEATURE_TYPE FeatureReader_4_<REJECTED> OUTPUT MyField FEATURE_TYPE FeatureReader_4_MyField' is not recognized by the factory 'QueryFactory'

 

I began digging in to this, but I'm stumped right now. I only began working with FME three weeks ago (with some support here and there from other experts), so I'm pretty lost when it comes to running via command line. I tried batch deployment also, but since there's so much to learn I'd rather post my issue here and get some guidance in the right direction.

Thanks for any support,

ML

 

icon

Best answer by mike707 7 July 2020, 16:07

View original

10 replies

redgeographics
Rank
Userlevel 4
Badge +25

One thing to try is this: run the workspace in FME Workbench and then look at the top of the log file, there's the proper command line syntax right there, you may be missing some user parameters if you just call the .fmw file (but it's hard to say without seeing your workspace).

david_r
Rank
Userlevel 4

You may also want to look at this article: https://knowledge.safe.com/articles/975/batch-processing-method-1-command-line-or-batch-fi.html

M

One thing to try is this: run the workspace in FME Workbench and then look at the top of the log file, there's the proper command line syntax right there, you may be missing some user parameters if you just call the .fmw file (but it's hard to say without seeing your workspace).

The "mother" Workbench runs with the following parameters (names redacted):

Command line arguments are `C:\Program Files\FME\2020.1\fme.exe' `C:/Users/User.Name/Path\wb-xlate-1593633503067_19464' `LOG_STANDARDOUT' `YES' `LogCountServerName' `{5444ef5d-3120-478a-814a-b24d618d2cf6}' `DEBUG_PORT' `61003' `LOG_TIMINGS' `Yes'

Note the "wb-xlate-1593633503067_19464". I believe it's the temporary file that is written when the process is ran. 

 

The command line I run to invoke the "mother" WB (and that fails) is simply:

fme "C:\Users\User.Name\Path\0_Run_Workbenches_in_sequence.fmw"

 

From what I see in the log when ran via FME Workbench, the parameters it includes are essentially calling the FMW file, some logging and debug parameters, and that's it. I don't see anything obvious there that could be missing when I call it.

But I'll check the other responses.

 

Since the "mother" workbench is in charge of calling the others, then I presume I only need to worry about the parameters for this one (which is the one called via command line).

I took a look at the logs and commands from the 4 "child" Workbenches, and here they are, just for information:

1)

INFORM|FME Configuration: Command line arguments are `C:\Program Files\FME\2020.1\fme.exe' `C:/Users/USer.Name/Path/1_Config.fmw' `--SourceDataset_JSON' `https://ADDRESS/?format=json&paremeters)' `--SourceDataset_JSON_3' `https://ADDRESS/?format=json=&parameters' `--SourceDataset_TEXTLINE' `C:\Users\User.Name\Pathl\configs/dummy.txt' `--FEATURE_TYPES' `' `--FME_LAUNCH_VIEWER_APP' `YES'

 

2)

FME Configuration: Command line arguments are `C:\Program Files\FME\2020.1\fme.exe' `C:/Users/User.Name/Path/2_config.fmw' `--Excel_Sheet' `C:\Users\User.Name\Path\My_Sheet.xlsx' `--SourceDataset_TEXTLINE' `C:\Users\User.Name\Path\configs/dummy.txt' `--FEATURE_TYPES' `' `--FME_LAUNCH_VIEWER_APP' `YES'

 

3)

INFORM|FME Configuration: Command line arguments are `C:\Program Files\FME\2020.1\fme.exe' `C:/Users/User.Name/path/3_config.fmw' `--DestDataset_CSV2' `C:\Users\User.Name/Path' `--SourceDataset_CSV2' `http://datasource' `--SourceDataset_TEXTLINE' `C:\Users\User.Name/Path\configs/dummy.txt' `--FEATURE_TYPES' `' `--FME_LAUNCH_VIEWER_APP' `YES'

 

4)

INFORM|FME Configuration: Command line arguments are `C:\Program Files\FME\2020.1\fme.exe' `C:/Users/User.Name/Path/4_config.fmw' `--SourceDataset_XLSXR' `C:\Users\User.Name\Path\Another_Sheet.xlsx' `--SourceDataset_TEXTLINE' `C:\Users\User.Name\Path\configs/dummy.txt' `--FEATURE_TYPES' `' `--FME_LAUNCH_VIEWER_APP' `YES'
M

You may also want to look at this article: https://knowledge.safe.com/articles/975/batch-processing-method-1-command-line-or-batch-fi.html

Thanks for the link. I had a look at the command line parameters, more specifically in to passing the data sets. But as the article mentions, the data sets are already defined in the workbenches themselves - they don't change.

Nevertheless, let me dig further in to this.

Thanks

david_r
Rank
Userlevel 4

Concerning WorkspaceRunners, be mindful about relative paths in the transformer settings and the current directory when running your workspace.

In your batch file, try something like:

cd "C:\Users\User.Name\Path\Projects Shared\Project_Name\configs"
"C:\Program Files\FME\fme.exe" 4_Workbench.fmw

Notice that we first change the working directory to the location where all the workspaces are located. 

M

SOLVED

I dug in to the logs and in to the valuable hints placed here, and found the reasons.

I began by splitting the workbench, to try and find exactly where the error was triggered, and I found two things:

  1. All of the Workbenches start with a dummy Text File Reader, that does nothing but triggers the transformers that follow (a System Caller to run a batch file, followed by the main transformers - mostly Feature Readers. This reader was causing the issue. I really didn't want/need the reader but it's the only way to trigger the transformers.
  2. The error in the aforementioned reader indicated a Python error. Looking in to the FME command line parameters, the INFO parameter revealed the root cause: the environmental variable for FME was/is pointing to FME 2017, which uses Python 2.7. I had the admin install 2020 for me but 2017 was not removed. I hadn't realized that the path was still pointing to 2017. Once I made the path to FME 2020 explicit in the command, it worked.

 

I feel a bit silly now, but happy that I found the cause.

 

I'm puzzled why the text file writers in the main WB, that output "success" or "failure" for each WB it calls, aren't doing anything. But I can live without that.

It's a bit of a bugger that if the WB uses Feature Readers (which are Transformers) they can't be ran on their own - some sort of caller is required to start the process.

 

Thanks!!

david_r
Rank
Userlevel 4

SOLVED

I dug in to the logs and in to the valuable hints placed here, and found the reasons.

I began by splitting the workbench, to try and find exactly where the error was triggered, and I found two things:

  1. All of the Workbenches start with a dummy Text File Reader, that does nothing but triggers the transformers that follow (a System Caller to run a batch file, followed by the main transformers - mostly Feature Readers. This reader was causing the issue. I really didn't want/need the reader but it's the only way to trigger the transformers.
  2. The error in the aforementioned reader indicated a Python error. Looking in to the FME command line parameters, the INFO parameter revealed the root cause: the environmental variable for FME was/is pointing to FME 2017, which uses Python 2.7. I had the admin install 2020 for me but 2017 was not removed. I hadn't realized that the path was still pointing to 2017. Once I made the path to FME 2020 explicit in the command, it worked.

 

I feel a bit silly now, but happy that I found the cause.

 

I'm puzzled why the text file writers in the main WB, that output "success" or "failure" for each WB it calls, aren't doing anything. But I can live without that.

It's a bit of a bugger that if the WB uses Feature Readers (which are Transformers) they can't be ran on their own - some sort of caller is required to start the process.

 

Thanks!!

Good to hear you found the solution and thanks for sharing.

Re the last comment, you can use the Creator transformer to trigger the FeatureReader.

ebygomm
Rank
Userlevel 1
Badge +10

SOLVED

I dug in to the logs and in to the valuable hints placed here, and found the reasons.

I began by splitting the workbench, to try and find exactly where the error was triggered, and I found two things:

  1. All of the Workbenches start with a dummy Text File Reader, that does nothing but triggers the transformers that follow (a System Caller to run a batch file, followed by the main transformers - mostly Feature Readers. This reader was causing the issue. I really didn't want/need the reader but it's the only way to trigger the transformers.
  2. The error in the aforementioned reader indicated a Python error. Looking in to the FME command line parameters, the INFO parameter revealed the root cause: the environmental variable for FME was/is pointing to FME 2017, which uses Python 2.7. I had the admin install 2020 for me but 2017 was not removed. I hadn't realized that the path was still pointing to 2017. Once I made the path to FME 2020 explicit in the command, it worked.

 

I feel a bit silly now, but happy that I found the cause.

 

I'm puzzled why the text file writers in the main WB, that output "success" or "failure" for each WB it calls, aren't doing anything. But I can live without that.

It's a bit of a bugger that if the WB uses Feature Readers (which are Transformers) they can't be ran on their own - some sort of caller is required to start the process.

 

Thanks!!

Is there a reason you cannot just use a Creator to trigger the transformers? Why do you need to read a text file?
M

Good to hear you found the solution and thanks for sharing.

Re the last comment, you can use the Creator transformer to trigger the FeatureReader.

Yes, but the reason I did that initially was because:

  1. I had 4 separate sequences in the same workbench. A proper reader would allow me to trigger them separately and also fully use the Run To / Run From tools
  2. I did in fact run a batch script first (but then removed it)

I left that for a while, but now I don't need the dummy readers anymore and the Creator works fine.

 

 

Thanks
M
Is there a reason you cannot just use a Creator to trigger the transformers? Why do you need to read a text file?

Yes, but the reason I did that initially was because:

  1. I had 4 separate sequences in the same workbench. A proper reader would allow me to trigger them separately and also fully use the Run To / Run From tools
  2. I did in fact run a batch script first (but then removed it)

I left that for a while, but now I don't need the dummy readers anymore and the Creator works fine.

 

 

Thanks

Reply


Community Stats

28,661
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.