I am creating a process that is going to move 3.5 million records from a SQL server db to a destination. The target can only be interacted with over SOAP. I am using a python caller to make the SOAP call because the HTTPCaller could not handle the challenge authentication method the target enforces.
I am batching the rows into groups of 20,000, creating the XML and passing that to the python caller as a single feature per batch. The calls become progressively slower throughout the run of the job. They start out around 4 seconds between batches and after 130 calls the time to process is around 130 seconds.
