Question

Run custom transformer end to end in a loop

  • 7 June 2018
  • 1 reply
  • 1 view

Badge

Hello,

My simplified flow is that I would like to take some features from DB. Next for each one do some processing using custom transformer (downloading for each some additional data etc.) and then save it to the zip file.

Is it possible to do it using custom transformer one by one I mean take one feature, then execute for it custom transformer, save results to zip and then take next one?

Right now it waits for all features to be processed and at some point I have out of memory exception :/

It would work if I would submit separate workspace from my main workspace to do that but it isn't the best solution for me.


1 reply

Badge +22

In your custom transformer publish the Parallel Process Group By, and then in your main canvas have a unique identifier on the feature (Counter if the data doesn't already have one) and set that attribute to the Group By.

 

 

Keep the parallel processing set to No Parallelism.

Reply