Today I cover a frequent issue: how to run FME from the command line, specifically with Windows Scheduler.
Question of the Week
This question was asked in Live Chat by a user with a problem with database connections and Windows Scheduler.
Q) In my workspace I use a writer that connects to SQL Server via a SQL Server account. In Workbench, the workspace runs and data is inserted. But run through Windows Task Scheduler, it fails with SQL Server issues.
I've set the username and password in the writer, but in the log file it appears to be not set!
A) This type of issue appears quite often. In short: if a workspace works in Workbench, but fails in a different environment, it's 95% likely to be a permissions issue.
Let's look at a few sticking points and - in particular - this scenario, which is harder because there's a database connection involved.
Who Are You?
The absolute number one issue when running a workspace from a command line is the account used to do that. It's important to realize that "you" (the user logged in) is different to the default account that Windows uses when it starts a process automatically.
For example, your account might have permission to read a specific file, while the Windows account used by Task Scheduler does not. In that scenario, the workspace fails to run:
ERROR | [Format] Reader: Failed to open file [Filename] for readingYou fix that by either setting the account that Task Scheduler will use, or by giving that Windows account the same privileges that you have. Here's an answer to a question asked last year:
"Fixed by ensuring the account used to run the script in Task Scheduler had Full Control permissions on the folder containing the script, the script itself, and any folders/files the script touches. Previously my batch file was in a dropbox location. I relocated it to C: drive and it works perfectly."
Back to this question, the user mentioned that:
"I'm using a domain user account with privileges to run the task on the windows machine."
So it looks like the Windows account is not directly the issue here...
Handling Connections
Database (and web) connections add a layer of complexity, because account permissions must be provided in FME.
This article outlines the steps involved. To quote its opening passage:
"The default for database and web connections is that connections stored in the default location are available to the current user only. If connections need to be accessible to any user in an organization, it is necessary to move both the connection storage file and the public key file to a shared location, and make the required connections public."
So step one is to put connections in a shared location available to all users. You can choose (in Tools > FME Options) whether each connection is private to you, or public to everyone. If the account running the task is different, then it needs to be public.
However, that may not be enough. The Windows default account also needs to know where the shared location is. You do that by setting the registry, as explained by this article
For this situation, the registry must be set manually in HKLM, which is where FME.exe looks if the keys are not present in HKLU.
In short, make your connections public and then - depending on which account is being used - set the registry to point to the public connections file.
I wouldn't - as this user did - embed connection information, unless you are sure the workspace file is secure.
Anyway, when the user in question found these articles, they were able to resolve the problem. But are there other issues?
Related Issues
Yes, there are plenty of other issues but - unsurprisingly - they're mostly related to accounts and permissions.
- In this question, the user found that a scheduled task needs the same process priority as the Workbench user.
- In this question, the user found that you need to run the process "in" the same folder as FME_HOME
- For some formats, FME needs access to ArcGIS functionality, and in this question the user discovered that the scheduler account may not see ArcGIS the same way.
- In this question, the user found that you have to be specific about which .exe file to use when multiple FME versions are installed.
- In this question, the user found you have to log into the service account at least once (to create the AppData folders) and that network drives need the UNC path set (presumably the Windows account has different drives).
- In this question, the user found that you don't need to run a BAT file in the Scheduler; you can use the FME Command-Line directly.
...and I suspect there are plenty more.
Won't Get Fooled Again
To reiterate, the key for running a workspace under Windows Task Scheduler is to ensure the user running the task (which IS NOT the same as you) has sufficient privileges and access rights.
This includes read/write permissions, access to connections, network drive access, elevated priority, and the ability to locate dependent applications such as ArcGIS.
If a scheduled FME process fails, then think: permissions, permissions, permissions!
Or switch to FME Server/Cloud, which is designed to automate solutions without these oddities!
Other Notable Questions
Let's see:
- Reading an Oracle View as a new feature type
- @whkarto asks if you can add an Oracle View as a feature type in Workbench. A view is like a pre-processed set of data - a "virtual table". My belief is you can read a view; I've even heard of manually adding a metadata entry for the view. You can't spatially index a view, but Oracle will recognize and use indexing on the underlying table, though you get warnings and errors.
- You might also get better results with a Materialized View, over a standard View.
- Excessive data returned in a StringSearcher regex
- User @jayqueue asks why their regex returns a longer string than expected. @ebygomm solves the issue by noticing that the regex is "greedy".
- For those not in the know, a greedy regex searches for as long a string as possible. So if I searched my name (Mark Ireland) for a string beginning with "a" and ending in "r", I believe a non-greedy search would return "ar", but a greedy search would return "ark Ir" - i.e. extends the result as far as it can.
