Question

Insufficient memory available -- error code was 2 | How to resolve this issue

  • 7 December 2022
  • 4 replies
  • 97 views
arthur_bazin
Rank
Badge +1

Hi,

I have this message during a translation : 

Insufficient memory available -- error code was 2

I have already read every thread on these memory troubles but none of them have a clear answer about that and how to resolve it.

The 3 answers were always : adding ram/stop others software/move the temp dir... 

 

My case : 

I have a machine with : 

  • 32Go of RAM 
  • TEMP_DIR set to a physical disk with 200Go of free space

 

I need to transfer 3 Go of data from one table in an Oracle database to a PostgreSQL database whitout any tranformation, it is just a reader + feature type connected to a writer + feature type.

 

On this machine, there is FME but there is the Oracle instance and the PostgreSQL instance too. They consumme some RAM :

  • About 10 Go for Oracle
  • About 100 Mo for PostgreSQL
  • Add 6 Go for other stuf mainly Windows process

 

So now I have only 15Go of free RAM (but 200Go of free physical space).

 

When I start the translation I have these messages :

System Status: 207.05 GB of disk space available in the FME temporary folder (E:\FME_TEMP\wbrun_1670401071313_19472\fmetmp_2)
System Status: 31.96 GB of physical memory available
System Status: 33.96 GB of virtual memory available
 
START - ProcessID: 19028, peak process memory usage: 44044 kB, current process memory usage: 44044 kB
 
FME Configuration: Start freeing memory when the process exceeds 95.87 GB
FME Configuration: Stop freeing memory when the process is below 71.90 GB
 
Failed to free sufficient memory to reach the process usage limit.
 
Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.

As you can see, FME does not detect that somme RAM is already taken and consider that all the RAM is available...

 

Is there any way to fix the amount of ram FME can use ?

Is there any real solution for this trouble that is a non sense for the amount of data I need to transfert...

 

Thank you very much for your help !

4 replies

M
Badge +2

@arthur_bazin​ FME shouldn't be staling on this task.

What version of FME are you using and are you running 32 or 64bit FME?

You mention that both of your databases are running on the same machine as FME. Remeber that as soon as FME make it's first data request to the databases they start consuming additional resources. Perhaps watch the process monitor.

Does the error tend to occur at the same point in your translation? Occasionally we have see an out of memory due to a bad geometry.

Try using FeatureWriter instead of the the conventional writer - although if you're only shifting one table I'm not sure if this will help.

As a last resort try splitting the data transfer. The easiest way is to use a BETWEEN on a sequential ID.

As a last reso

arthur_bazin
Rank
Badge +1

Hi,

Thanks for your answer, I will give more info.

The version : FME(R) 2021.2.1.0 (20211203 - Build 21797 - WIN64)

An extract of the log to have an idea of the timing and what is going on : 

2022-12-08 08:58:14|   0.3|  0.2|INFORM|Creating writer for format: 
2022-12-08 08:58:14|   0.3|  0.0|INFORM|Creating reader for format: 
2022-12-08 08:58:14|   0.3|  0.0|INFORM|MULTI_READER(MULTI_READER): Will fail with first member reader failure
2022-12-08 08:58:14|   0.3|  0.0|INFORM|MULTI_READER(MULTI_READER): Adding ORACLE_NONSPATIAL Reader with keyword ORACLE_NONSPATIAL_1
2022-12-08 08:58:14|   0.3|  0.0|INFORM|Using Multi Reader with keyword `MULTI_READER' to read multiple datasets
2022-12-08 08:58:15|   0.5|  0.2|INFORM|FME Configuration: Using connection values from '127.0.0.1 - Oracle 12c - BB1' for ORACLE_NONSPATIAL_1
2022-12-08 08:58:15|   0.5|  0.0|INFORM|Creating reader for format: Oracle Non-Spatial
2022-12-08 08:58:15|   0.7|  0.2|INFORM|ORACLE Reader: Using Oracle Reader to read tables from database `', server type `ORACLE8i', server name `BDD01', user name `SYSTEM', password `***'
2022-12-08 08:58:15|   0.7|  0.0|INFORM|ORACLE Reader: Using Rich geometry.
2022-12-08 08:58:15|   0.7|  0.0|INFORM|Loaded module 'OCI' from file 'OCI.dll'
2022-12-08 08:58:15|   0.7|  0.0|INFORM|Environment variable ORACLE_HOME has value `C:\app\oracle\product\12c\db_1'
2022-12-08 08:58:15|   0.7|  0.0|INFORM|Connecting to ORACLE database `BDD01' with user name of `SYSTEM' and password of `***'
2022-12-08 08:58:15|   0.8|  0.1|INFORM|Connected to ORACLE database `BDD01'
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Environment variable ORACLE_HOME has value `C:\app\oracle\product\12c\db_1'
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Connecting to ORACLE database `BDD01' with user name of `SYSTEM' and password of `***'
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Connected to ORACLE database `BDD01'
2022-12-08 08:58:15|   0.8|  0.0|INFORM|ORACLE_NONSPATIAL reader: Creating bulk features from individual features
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Using MultiWriter with keyword `MULTI_WRITER' to output data (ID_ATTRIBUTE is `multi_writer_id')
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Writer output will be ordered by value of multi_writer_id
2022-12-08 08:58:15|   0.8|  0.0|INFORM|Loaded module 'LogCount_func' from file 'C:\Program Files\FME\plugins/LogCount_func.dll'
2022-12-08 08:58:15|   0.8|  0.0|INFORM|FME API version of module 'LogCount_func' matches current internal version (3.8 20210302)
Reading...
2022-12-08 08:58:15|   0.9|  0.1|INFORM|ORACLE Reader: Executing SQL Statement `SELECT "PROJET_xxxx_BD_AC"."MEDIA"."MEDIA", "PROJET_xxxx_BD_AC"."MEDIA"."MINIATURE", "PROJET_xxxx_BD_AC"."MEDIA"."NOM_DU_FICHIER", "PROJET_xxxx_BD_AC"."MEDIA"."TYPE", "PROJET_xxxx_BD_AC"."MEDIA"."ID_MEDIA", "PROJET_xxxx_BD_AC"."MEDIA"."AUTEUR", "PROJET_xxxx_BD_AC"."MEDIA"."DESCRIPTION", "PROJET_xxxx_BD_AC"."MEDIA"."TITRE", "PROJET_xxxx_BD_AC"."MEDIA"."DATE_AJOUT", "PROJET_xxxx_BD_AC"."MEDIA"."DATE_MODIF", "PROJET_xxxx_BD_AC"."MEDIA"."BG_COMMENTAIRE", "PROJET_xxxx_BD_AC"."MEDIA"."MEDIA_FICHIER", "PROJET_xxxx_BD_AC"."MEDIA"."MEDIA_FICHIER_ORIGINE" FROM "PROJET_xxxx_BD_AC"."MEDIA"' to obtain features for feature type `PROJET_xxxx_BD_AC.MEDIA'
2022-12-08 09:01:50|  22.3| 21.4|WARN  |Failed to free sufficient memory to reach the process usage limit. To improve stability and performance please increase the memory available to FME. Available memory can be increased by adding physical RAM, increasing swap space, or closing other applications
2022-12-08 09:05:05|  38.3| 16.0|INFORM|FME Configuration: Using connection values from '1 - Local - PG 11 - bazinga' for POSTGRES_1
2022-12-08 09:05:05|  38.3|  0.0|INFORM|Creating writer for format: PostgreSQL
2022-12-08 09:05:05|  38.5|  0.2|INFORM|Trying to find a DYNAMIC plugin for writer named `POSTGRES'
2022-12-08 09:05:05|  38.5|  0.0|INFORM|Loaded module 'POSTGRES' from file 'C:\Program Files\FME\plugins/POSTGIS.dll'
2022-12-08 09:05:05|  38.5|  0.0|INFORM|FME API version of module 'POSTGRES' matches current internal version (3.8 20210302)
2022-12-08 09:05:05|  38.5|  0.0|INFORM|FME Configuration: No destination coordinate system set
2022-12-08 09:05:05|  38.5|  0.0|INFORM|POSTGRES writer: Splitting bulk features into individual features
2022-12-08 09:05:14|  47.7|  9.2|FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.
Out of memory.  Program terminating.  Please read the FME Help section 'Adjusting Memory Resources' for workarounds.
Warning: not all FMESessions that were created were destroyed before shutdown.  This may cause instability

Some info of the machine : 

  • Before translation
    • PROC : 10%
    • RAM : 16,2/32 Go
    • Oracle 10018 Mo
    • FME : 234 Mo
    • PostgreSQL : 15 Mo
  • 2 min after the start of the translation :
    • Oracle : 10020 Mo
    • FME : 3147 Mo
    • PostgreSQL : 15 Mo
  • When it crashes :
    • Proc : 10%
    • RAM : 24.1/32Go
    • Oracle : 10018 Mo
    • FME : 8400,3 Mo
    • PostgreSQL : 15 Mo

In fact, the processes of PostgreSQL (7 processes) have no change in their ressources consumption. I think that FME haven't had the time to contact PostgreSQL.

 

 

M
Badge +2

@arthur_bazin​ Definitely very odd behaviour. I think I would take this to your local FME provider as they will be able to look more closely at your Oracle data and try and determine what is causing the memory issue.

In the meantime you could try:

  • JDBC driver for Oracle
  • OBDC driver for Oracle
  • SQLExecutor with a direct query.
arthur_bazin
Rank
Badge +1

@arthur_bazin​ Definitely very odd behaviour. I think I would take this to your local FME provider as they will be able to look more closely at your Oracle data and try and determine what is causing the memory issue.

In the meantime you could try:

  • JDBC driver for Oracle
  • OBDC driver for Oracle
  • SQLExecutor with a direct query.

Sorry for my late reply...

 

Thanks for the tip, I will try these driver/transformers and keep you informed !

Reply


Community Stats

28,666
Posts
109,339
Replies
36,535
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.