Question

Error running translation when writing to a database

M

I am converting over 1.5 million links and writing them to a database. This workbench has worked for me previously with roughly the same amount of data. Since updating to FME 2019 I am now getting an "Error running translation" with nothing to really go on. I enabled full logging and these are the last few lines:

2019-08-20 09:49:51|5835.7|  0.0|WARN  |ResourceManager: Failed to free sufficient memory to reach the upper memory limit: Physical 1.85 GB over 917.49 MB Virtual 7.34 GB over 16777216.00 TB
2019-08-20 09:49:51|5835.8|  0.2|WARN  |ResourceManager: Freeing! Over private memory limit: 1.85 GB > 917.49 MB
2019-08-20 09:49:51|5835.8|  0.0|INFORM|ResourceManager: Optimizing Memory Usage.  Please wait...
2019-08-20 09:49:51|5835.9|  0.0|WARN  |ResourceManager: Freed 906451 bytes of vector data (memory decreased by 196608 bytes)
2019-08-20 09:49:51|5835.9|  0.0|WARN  |ResourceManager: Failed to free sufficient memory to reach the upper memory limit: Physical 1.85 GB over 917.49 MB Virtual 7.34 GB over 16777216.00 TB
2019-08-20 09:49:52|5836.1|  0.2|WARN  |ResourceManager: Freeing! Over private memory limit: 1.86 GB > 917.49 MB
2019-08-20 09:49:52|5836.1|  0.0|INFORM|ResourceManager: Optimizing Memory Usage.  Please wait...
2019-08-20 09:49:52|5836.1|  0.0|WARN  |ResourceManager: Freed 339456 bytes of vector data (memory decreased by 131072 bytes)
2019-08-20 09:49:52|5836.1|  0.0|WARN  |ResourceManager: Failed to free sufficient memory to reach the upper memory limit: Physical 1.86 GB over 917.49 MB Virtual 7.34 GB over 16777216.00 TB
2019-08-20 09:49:52|5836.2|  0.2|WARN  |ResourceManager: Freeing! Over private memory limit: 1.86 GB > 917.49 MB
2019-08-20 09:49:52|5836.2|  0.0|INFORM|ResourceManager: Optimizing Memory Usage.  Please wait...
2019-08-20 09:49:52|5836.3|  0.0|WARN  |ResourceManager: Freed 1276748 bytes of vector data (memory decreased by 196608 bytes)
2019-08-20 09:49:52|5836.3|  0.0|WARN  |ResourceManager: Failed to free sufficient memory to reach the upper memory limit: Physical 1.86 GB over 917.49 MB Virtual 7.34 GB over 16777216.00 TB
2019-08-20 09:49:52|5836.4|  0.2|WARN  |ResourceManager: Freeing! Over private memory limit: 1.86 GB > 917.49 MB
2019-08-20 09:49:52|5836.4|  0.0|INFORM|ResourceManager: Optimizing Memory Usage.  Please wait...

Any ideas on how I can fix this? It appears to be a memory issue but not sure what can be done. The workbook takes in line data, creates lines out of them and joins a few other tables using a feature joiner to create our model.

10 replies

david_r
Rank
Userlevel 4

How much physical RAM do you have available on the server? Are you using 32-bit or 64-bit FME?

My best tip is to make sure to have a lot of available physical RAM (not virtual memory) and use 64-bit FME.

M

How much physical RAM do you have available on the server? Are you using 32-bit or 64-bit FME?

My best tip is to make sure to have a lot of available physical RAM (not virtual memory) and use 64-bit FME.

The server is a localhost MSSQL on my machine. I have 32GB RAM on my desktop and I am using 64 bit FME

david_r
Rank
Userlevel 4

The server is a localhost MSSQL on my machine. I have 32GB RAM on my desktop and I am using 64 bit FME

And how much of the 32GB RAM is currently available?

From the command line, type "systeminfo" and look in the output, for example:

Available Physical Memory: 2'144 MB

Also have a look at the discussion here on how to tell FME how much RAM to use: https://knowledge.safe.com/idea/40283/parameter-to-set-maximum-memory-usage-for-64bit.html

M

And how much of the 32GB RAM is currently available?

From the command line, type "systeminfo" and look in the output, for example:

Available Physical Memory: 2'144 MB

Also have a look at the discussion here on how to tell FME how much RAM to use: https://knowledge.safe.com/idea/40283/parameter-to-set-maximum-memory-usage-for-64bit.html

Available Physical Memory: 7,928 MB

david_r
Rank
Userlevel 4

Available Physical Memory: 7,928 MB

Unless someone else has an idea, I'd recommend that you contact your FME reseller for support through Safe. They will probably ask for a copy of your workspace and a complete log file.

M

Turning bulk insert to "No" allowed the translation to work without failing. Not sure if this was just a fluke or if it actually made a difference to the memory problem?

david_r
Rank
Userlevel 4

Turning bulk insert to "No" allowed the translation to work without failing. Not sure if this was just a fluke or if it actually made a difference to the memory problem?

Only one way to find out, try burning it back on again :-)

virtualcitymatt
Rank
Userlevel 4
Badge +26

I've noticed that if you have a lot of Applications open when you start a workspace then FME seems to take what is available at the time and doesn't get more.

 

 

If you look at the first several lines on you're logfile you should see something like around line 35:

 

 

|INFORM|FME Configuration: Process limit is 15.44 GB of physical memory
|INFORM|FME Configuration: Start freeing memory when process usage exceeds 46.32 GB of virtual memory
|INFORM|FME Configuration: Stop freeing memory when process usage is below 34.74 GB of virtual memory

 

The numbers represent (in order)

 

1. Your physical RAM

2. The maximum amount of 'memory' FME will consume before it starts dumping memory to the temp directory

3. When FME will start filling up the 'memory' again.

 

 

When you see the line  ResourceManager: Optimizing Memory Usage.  Please wait... FME is trying to dump data to to the hard disk. As I understand it FME needs to keep some in memory so sometimes it can't free it all.

 

 

when you have a lot of apps onpen sometimes you might see something a lot lower at the start of your log: 
|INFORM|FME Configuration: Process limit is 15.81 GB of physical memory
|INFORM|FME Configuration: Start freeing memory when process usage exceeds 33.22 MB of virtual memory
|INFORM|FME Configuration: Stop freeing memory when process usage is below 31.89 MB of virtual memory

Here you can see FME will start dumping much lower, this will remain until the workspace is finished and will take a lot longer to finish and could cause some real issues with the process

 

One other thing to try is to run FME as administrator. Sometimes I've seen problems when fme has issues writing to the temp directory for what ever reason. Running as Admin can sometimes overcome this strange issue (I would recommend a reinstall id this fixes your issue).

 

 

S

I would just like to add that I was experiencing the same issue after updating from FME 2018 to FME 2019 on a workspace that worked flawlessly before. It sometimes showed this error message, but not always. The workspace always seemed to stop after a neighbourfinder, but would work with a smaller dataset, so must have been having a memory issue (even though my machine is pretty beefy). I had made a workaround for the FME 2019 version by first writing all the raw data (multiple CAD files) to a single file, then running the workbench using a reader pointed to that file.

 

After upgrading to FME 2020 the problem has disappeared and I no longer need to use the workaround.

virtualcitymatt
Rank
Userlevel 4
Badge +26

I would just like to add that I was experiencing the same issue after updating from FME 2018 to FME 2019 on a workspace that worked flawlessly before. It sometimes showed this error message, but not always. The workspace always seemed to stop after a neighbourfinder, but would work with a smaller dataset, so must have been having a memory issue (even though my machine is pretty beefy). I had made a workaround for the FME 2019 version by first writing all the raw data (multiple CAD files) to a single file, then running the workbench using a reader pointed to that file.

 

After upgrading to FME 2020 the problem has disappeared and I no longer need to use the workaround.

Awesome to know!

Reply


Community Stats

28,707
Posts
109,472
Replies
36,550
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.