Skip to main content
  • EN

I have a workspace that reads from a very large table in MSSQL, unpivots the data and does some minor/simple processing in FME, then writes the data out to postgres. This workspace takes around 3 - 3.5 days to finish on FME Server. Reading the data doesn't take long, but processing and writing does.

 

In an attempt to speed up the process, I tried to apply parallel processing by wrapping the entire process in a custom transformer with FeatureReaders and FeatureWriters instead of 'native' readers and writers to concurrently read different sections of the table, then process and write the data. I wanted to test the performance on FME Server with the writers turned off and minimal parallel processing (4 child processes) but it fails rather quickly with the following error:

 

3682021-12-13 14:55:51 | Error in child process '1', exited with code '1'

3692021-12-13 14:55:51 | f_38 (TransformFact): Error in child process '1', exited with code '1'

 

The log file doesn't tell me much other than the following vague message:

 

3632021-12-13 14:55:50 | 1> An error occurred. FME will attempt to obtain more information on the error, but this may cause the translation to be terminated

3642021-12-13 14:55:50 | 1> Successfully closed POSTGRES database reader

 

I'm not sure what is causing this error or where to start. With parallel processing disabled in the custom transformer, I am able to test the workspace just fine on FME Server. My first question is perhaps am I using parallel processing in a way that it wasn't intended for?

That's not a very useful error message indeed. It does look like parallel processing is causing this.

 

One thing I'm wondering about is whether or not database transactions / commits from the parallel processes are getting in eachother's way. Like one process can be doing a commit while another one is still writing.

That's not a very useful error message indeed. It does look like parallel processing is causing this.

 

One thing I'm wondering about is whether or not database transactions / commits from the parallel processes are getting in eachother's way. Like one process can be doing a commit while another one is still writing.

Unfortunately the error isn't helpful 😅 . The writer has been disabled during these tests on FME Server; I was just trying to test the performance of the transformers to see if there was any benefit. I'll add that the concurrent database transactions don't seem to be an issue (yet) as I was able to run the custom transformer with parallel processing enabled to write a small subset of the data to a staging table, unfortunately I was only able to do this on FME Desktop and not Server.

I was able to isolate the problem. I had a second custom transformer encapsulated in the first custom transformer that was causing the error, although I'm not sure why. It wasn't doing anything special. The workspace works as expected when I remove these custom transformers and take everything out instead.

Reply


Community Stats

30,171
Posts
114,532
Replies
37,828
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.