Skip to main content
  • EN

I am creating a process that is going to move 3.5 million records from a SQL server db to a destination. The target can only be interacted with over SOAP. I am using a python caller to make the SOAP call because the HTTPCaller could not handle the challenge authentication method the target enforces.

 

I am batching the rows into groups of 20,000, creating the XML and passing that to the python caller as a single feature per batch. The calls become progressively slower throughout the run of the job. They start out around 4 seconds between batches and after 130 calls the time to process is around 130 seconds.

 

 

My recommendation would be to disable all transformers downstream from the PythonCaller before testing. That way you can be sure if it's the PythonCaller that's the problem, or not. Also make sure that the PythonCaller does not accumulate all the groups in memory as the workspace progresses, but that it only retains one single group at a time.

Reply


Community Stats

30,187
Posts
114,578
Replies
37,853
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.