Hi all, I need some advice for 2 issues in FME server, which are possibly related:
1. Memory Issue (Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.):
I have an issue with a workbench I’ve made in FME desktop which behaves differently when run on FME server. Both environments have the same memory constraints.
The issue occurs when I try to load a handful (<5) of files into my workspace, which generally can handle upwards of 400 regular files. The files themselves aren’t much different (no odd characters that might trip the process up), but FME seems to get stuck on a subset of them, and keeps churning until all memory is used up. As mentioned, this only happens in FME Server.
On failing a Job due to no memory, it tries again: (Failed to free sufficient memory to reach the process usage limit. To improve stability and performance please increase the memory available to FME. Available memory can be increased by adding physical RAM, increasing swap space, or closing other applications).
Access Issue (Failed to open FME Feature store file):
A handful of files (out of >1000) cause the workspace to error out:
‘Failed to open FME Feature store file’ (see attached screenshot)
Happy to give more relevant information and logs.
Addressing the attachment, that suggests that the account running the FME Engines does not have sufficient permissions to write to the FME temporary directory. That path isn’t a valid Windows path, but is the Linux Root dir, are you running on Linux or Windows?
Sorry, a wall of questions…
- What version is Flow and Form
- What are the specs of the machine running Flow/Server?
- How much RAM is installed, how much is free?
- Number of CPU cores?
- Have you specified a custom temp directory?
- How many engines are configured to be running?
- What other jobs are running at the same time?
- Is the machine being used to run other software?
- How big are these files? What are they?
- What are you doing with them? Just reading the path, or parsing the data they contain
Hi,
Thanks for the detailed reply, responses below. The issue only appears sometimes. I’ve run hundreds of files through FME server without issue (200-300 at a time), but occasionally some get these errors.
- Dekstop Version: FME(R) 2022.0.1.1 (20220617 - Build 22350 - WIN64)
- Server Version: FME(R) 2022.0.0.2 Build 22343 - Win 64
Machine specs:
Dekstop:
System Status: 683.68 GB of disk space available in the FME temporary folder (\\prd.xxx01.ffm.Z\appdata\ReportSource\FME\Temp\wbrun_1723946054902_21576\fmetmp_1)
) - this is a network location
System Status: 32.00 GB of physical memory available
System Status: 128.00 GB of virtual memory available
Server:
System Status: 67.03 GB of disk space available in the FME temporary folder (C:\Windows\TEMP\fmeengines\XYZ_Engine1)
System Status: 16.00 GB of physical memory available
System Status: 64.00 GB of virtual memory available
I’m not sure how I can get other specs for the server computer, for cpu/gpu - an FME engine is being used.
- Temp directory: Above, No temp directory is set
- Engines configures: 2 engines on the server, 1 is configured to run for this process.
- Other Jobs: There are other jobs, but its all queued up
- Is the machine being used to run other software? Occasionally, but I’ve tried loading at hours where its not in use at all.
- The files themselves are tiny. All data files are generally <1mb, and 80pc of them are <200kb.
They are data readouts from a sensor. Each data file has a run of header + data, with many runs in the same .TXT. We process them by splitting the headers and the data and checking if the counterdata is valid (simple regex to ensure it looks like what is expected). There is also an archival component to the workbench, as well as a component to insert it into relevant database tables.
The thing that is mostly concerning is that I can easily run many big files through and not hit memory issues on FME server, but occasional, just a handful will cause it to get the error. There doesn’t seem to be anything different between these files.
Only thing that concerns me with the above is that you’re running a newer version of Form than whats on Flow. That’s not recommended as the newer Form may have components that are not present in Flow.
I would also look to setting your temporary directory as per: https://support.safe.com/hc/en-us/articles/25407446479373-Setting-a-temporary-file-location-for-FME-to-use-via-the-FME-TEMP-environment-variable
Other than that, it might be worth reaching out to your reseller to have a better look at your specific scenario