Hi @mferwerda111, in the first workspace are you specifying where to write the data? If so, what does that look like?
I can’t think of a setting off the top of my head, but as a workaround could you replace the drive path with a unc path and then use a featurereader that will always read the data in from a text file from the core machine.
Based on your wording it seems like you're running distributed engines without a common server system share, is that right?
If so, I suspect the solution is to rethink your installation so that your engines all connect to the same system share, see https://docs.safe.com/fme/html/FME_Server_Documentation/Content/AdminGuide/Install-Core-and-Engines-3-Tier.htm
So when the fmw runs, it creates a file, this file is automatically
created with ta guid name like “33358624659273_gpas_status.txt”
Example
{
"workspace": "GPAS_WSR.fmw",
"timeRequested": "Tue-05-Dec-2017
01:00:00 AM",
"NumFeaturesOutput": "250",
"requestKeyword":
"SERVER_SCHEDULER",
"timeStarted": "Tue-05-Dec-2017
01:00:00 AM",
"repository": "ETL_PROD",
"jobsuccess_topic":
"gPasDataLoadingStatus",
"LogFileName": "job_603.log",
"StatusMessage": "Translation
Successful",
"jobfailure_topic":
"gPasDataLoadingStatus",
"StatusNumber": "0",
"id": "603",
"timeFinished": "Tue-05-Dec-2017
01:00:21 AM",
"logHome":
"//ETL/FME16Repository///resources/logs",
"logUrl": "http://Nz1320/fmerest/v3/transformations/jobs/id/603/log"
}
We have FME server set up so it uses a common drive, all the repository’s,
resources, etc, are on the same drive where all the engines can see the same
data like \\\\my.net\\Reference\\ETL\\FME16Repository\\
In our old set up, these files were automatically created in this
path, and all the engines can get to them.
\\\\myfolder\\Reference\\GeoTool\\EDW\\FME16Repository\\resources\\system\\temp\\requestdata\\4072616362113062_gpas_status.txt'
In our new install, for some reason, the files are being created
here on the host core box, so the fmw run on distributed engines cannot see
it. The process is passing he correct
path the the called fmw, so that part is technically working.
e:\\FMEServer\\Utilities\\tomcat\\temp\\33358624659273_gpas_status.txt
VDC
Somewhere there is something that is telling FME server where to
create these files
I would start by looking into the configuration files, notably if you find any of these in your engine installation:
- fmeCommonConfig.txt
- fmeEngineConfig.txt
- fmeServerConfig.txt
In particular, see if you can find a reference to your tomcat temp folder somewhere there.
I would start by looking into the configuration files, notably if you find any of these in your engine installation:
- fmeCommonConfig.txt
- fmeEngineConfig.txt
- fmeServerConfig.txt
In particular, see if you can find a reference to your tomcat temp folder somewhere there.
In fmeEngineConfig.txt, check your definition of the FME_SHAREDRESOURCE_* macros (probably on lines 18-23), they should all point at your shared server repository.
Also check the following values in <FMEServer>\\Server\\config\\subscribers\\workspace.properties
- TOPIC_DATA_FOLDER
- RESULT_DATA_FOLDER
For some reason they are both relative paths on my (single engine) installation, which is always a bit suspicious when this type of error occurs. You could always try to set them to absolute paths and see if that makes a difference. Just don't forget to restart your core+engine service if you change anything.
Can you confirm that the instructions for
https://docs.safe.com/fme/html/FME_Server_Documentation/Content/AdminGuide/Adding_FME_Engines_on_Separate_Machine.htm were followed?
I'm wondering if you look at step 10, what the REQUEST_DATA_DIR parameter is set to for each service?
Also check the following values in <FMEServer>\\Server\\config\\subscribers\\workspace.properties
- TOPIC_DATA_FOLDER
- RESULT_DATA_FOLDER
For some reason they are both relative paths on my (single engine) installation, which is always a bit suspicious when this type of error occurs. You could always try to set them to absolute paths and see if that makes a difference. Just don't forget to restart your core+engine service if you change anything.
I have looked over the config files and the workspace.properties and do not see anything obvious yet. We do have a install of FME server in our DEV environment, that is supposed to mimic the prod install, and this process is working correctly in that install, so I have been comparing the config files in that install to the prod config files, but still see nothing different other than what one would expect
So we do have a install of FME server in our DEV environment, that is supposed to mimic the prod install, and this process is working correctly in that install.
I have looked at the E:/FMEServer\\Utilities\\tomcat\\webapps\\fmejobsubmitter\\WEB-INF\\conf\\propertiesFile.properties
and see this, which looks OK to me
fmejobsubmitter
#REQUEST_DATA_DIR - The directory which stores request data from HTTP POST body
REQUEST_DATA_DIR=//usda.net/fs/FS/Reference/ETL/FME16Repository//resources/system/temp/requestdata
fmenotification
#REQUEST_DATA_DIR - The directory which stores request data from HTTP POST body
REQUEST_DATA_DIR=//usda.net/fs/FS/Reference/ETL/FME16Repository//resources/system/temp/requestdata
Status update, so far I have not found anything obvious in config files, it is probably in there someplace. I have implemented a work around and modified the Target URL for the subscription to have a job tag so it only runs on the core, and then passes the json text to another fmw that will do the writing (we do not have oracle client on the core host)
I think it might be good to add in a parameter for Job tag when registering with the notification service
Hi @mferwerda111– I'm really sorry to read about this! I reviewed this Q&A; posting (incl. its comments) and I couldn't find explicit mention, so to confirm: both FME Server installs (Dev/Prod) are the same build?
I wonder if there might be a system environment variable affecting the Prod installation? Can you confirm if this might be the issue or not?
Hi @mferwerda111– I'm really sorry to read about this! I reviewed this Q&A; posting (incl. its comments) and I couldn't find explicit mention, so to confirm: both FME Server installs (Dev/Prod) are the same build?
I wonder if there might be a system environment variable affecting the Prod installation? Can you confirm if this might be the issue or not?
Yes, both are FME Server 2016.1.2.1 - Build 16674 - win64
So this worked in our old environment, works in a new dev environment, but not the prod install, both the Dev FME Server and Prod FME server are in the same "environment" or at least it is supposed to be. I use the back up and restore process to move all the content over from the Old environment to both the Dev FME Server and prod FME Server in the new environment. The PATH Environment Variables for FME server look similar in both, slightly different paths,but basically the same.
Is there another Env Var I should look at ?
Yes, both are FME Server 2016.1.2.1 - Build 16674 - win64
So this worked in our old environment, works in a new dev environment, but not the prod install, both the Dev FME Server and Prod FME server are in the same "environment" or at least it is supposed to be. I use the back up and restore process to move all the content over from the Old environment to both the Dev FME Server and prod FME Server in the new environment. The PATH Environment Variables for FME server look similar in both, slightly different paths,but basically the same.
Is there another Env Var I should look at ?
With respect to environment variables, there isn't any one in particular I had in mind – essentially just a diff between the two machines. This issue has me puzzled.
With respect to environment variables, there isn't any one in particular I had in mind – essentially just a diff between the two machines. This issue has me puzzled.
I think Jennifer might have been on the right track, I am looking at the Utilities\\tomcat\\webapps\\fmenotification\\WEB-INF\\conf\\propertiesFile.properties
file and I see this path:
#REQUEST_DATA_DIR - The directory which stores request data
from HTTP POST body
REQUEST_DATA_DIR=//my.net/Ref/ETL/FME16Repository//resources/system/temp/requestdata
Not sure why there are a // in there, and I can see that the requestdata folder exists in our old install, and in our dev install, but that folder does not exist in our prod install.
I have a feeling that // is the problem
I think Jennifer might have been on the right track, I am looking at the Utilities\\tomcat\\webapps\\fmenotification\\WEB-INF\\conf\\propertiesFile.properties
file and I see this path:
#REQUEST_DATA_DIR - The directory which stores request data
from HTTP POST body
REQUEST_DATA_DIR=//my.net/Ref/ETL/FME16Repository//resources/system/temp/requestdata
Not sure why there are a // in there, and I can see that the requestdata folder exists in our old install, and in our dev install, but that folder does not exist in our prod install.
I have a feeling that // is the problem
or I think that was david_r suggested looking at those files
or I think that was david_r suggested looking at those files
I would be surprised if a double slash was the issue here. They're usually ignored by the file system/OS.
I would be surprised if a double slash was the issue here. They're usually ignored by the file system/OS.
Yes, the // is not the issue. We manually added the requestdata folder and now it is working as expected, not sure why that folder did not get created. And thanks for the help pointing to those config files which eventually got me to look to see if the folder was actually there.
