Skip to main content
  • EN

Hi guys,

We have a routine job that reads 5 million records from a Oracle non-spatial table (runs on fmeserver, 2017). During a recent run it completed successfully but only processed 2 million records. The log looks normal, it is as if the source db only had 2 million records - which is not the case when checked. Normally we would get a oracle/connection error if there was a network failure - the job does take a while to run. Has anyone seen behaviour like this before? Thanks!

Grace

Have you also checked the server logs for warnings or errors?

Most importantly check fmeserver.log and fmeprocessmonitorengine.log

Hi @david_r. We have 5 engines, I'm not sure how to differentiate between them reading the server logs. The following messages repeated leading up to the job completing. The error "Invalid response length returned..." started to appear about 5 min before the job completed. Not sure what to make of that...

Invalid response length returned by FME Engine, expected 369,295,873 but read 511 bytes

 

Could not read FME Engine name. Registration cancelled.

 

Disconnected FME Engine: on host

Failed to register FME Engine.

 

Accepted new FME Engine connection.

 

Registering FME Engine...

 

Invalid response length returned by FME Engine, expected 369,296,131 but read 807 bytes

 

Could not read FME Engine name. Registration cancelled.

 

Accepted new client connection

 

Successful login by user admin

 

Accepted new client connection

 

Successful login by user admin

 

Accepted new FME Engine connection.

 

Registering FME Engine...

 

FME Server license does not allow more than maximum of 5 FME Engine(s).

 

Job 6,136: Pulled from result queue for delivery.

 

Job 6136: FME response successully processed

Hi @lc

 

 

Have you seen this article?

 

A few things stood out here that may apply to your situation:
  • oracle client configuration errors
  • large datasets with memory intensive operations

     

     

Do you get the same behaviour if you use the Oracle reader instead of a FeatureReader?

 

 

Is there any way you can read and process smaller chunks over multiple jobs and see if the problem persists?

Hi @lc

 

 

Have you seen this article?

 

A few things stood out here that may apply to your situation:
  • oracle client configuration errors
  • large datasets with memory intensive operations

     

     

Do you get the same behaviour if you use the Oracle reader instead of a FeatureReader?

 

 

Is there any way you can read and process smaller chunks over multiple jobs and see if the problem persists?

Hmm i don't see those errors replicated in my log. i don't see anything pointing to a network or memory issue. we've run this job successfully before, a number of times. Updating the reader or splitting the job would require significant changes to a complex process. would like to avoid if possible.

 

 

Hi @david_r. We have 5 engines, I'm not sure how to differentiate between them reading the server logs. The following messages repeated leading up to the job completing. The error "Invalid response length returned..." started to appear about 5 min before the job completed. Not sure what to make of that...

Invalid response length returned by FME Engine, expected 369,295,873 but read 511 bytes

 

Could not read FME Engine name. Registration cancelled.

 

Disconnected FME Engine: on host

Failed to register FME Engine.

 

Accepted new FME Engine connection.

 

Registering FME Engine...

 

Invalid response length returned by FME Engine, expected 369,296,131 but read 807 bytes

 

Could not read FME Engine name. Registration cancelled.

 

Accepted new client connection

 

Successful login by user admin

 

Accepted new client connection

 

Successful login by user admin

 

Accepted new FME Engine connection.

 

Registering FME Engine...

 

FME Server license does not allow more than maximum of 5 FME Engine(s).

 

Job 6,136: Pulled from result queue for delivery.

 

Job 6136: FME response successully processed
I don't think that's the issue. I'd primarily look for ORA-xxxx errors etc.
Hmm i don't see those errors replicated in my log. i don't see anything pointing to a network or memory issue. we've run this job successfully before, a number of times. Updating the reader or splitting the job would require significant changes to a complex process. would like to avoid if possible.

 

 

You may also want to try creating a super simple workspace that does nothing else than read the Oracle table and send everything to the NULL writer. I'm curious to know if that works or if you get the same error.
You may also want to try creating a super simple workspace that does nothing else than read the Oracle table and send everything to the NULL writer. I'm curious to know if that works or if you get the same error.
NULL writer? Learned something new today, thanks.
NULL writer? Learned something new today, thanks.
Surprisingly handy sometimes! 😃

How long is the process running? It could be that it takes so much time that FME loses connection with your database?

Reply


Community Stats

30,159
Posts
114,480
Replies
37,806
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.