Solved

Slow performance speed in workbench containing Custom Looping Transformer

  • 20 February 2018
  • 5 replies
  • 17 views

Badge +1

Hi,

I have a FME 2016.1 workbench that contains a Custom Loop transformer and within that I also have a PythonCaller with some Python Code.

The workbench outputs the expected result but it takes over an hour to run. I need it to run ideally in less than 20 minutes.

I have attached some screen shots of the main workbench and the custom transformer. Does anyone have ideas on things I could do to speed this up?

Thanks,

icon

Best answer by gio 21 February 2018, 15:25

View original

5 replies

Badge
Most likely something to do with your custom loop transformer, without test data and workbench won't be able to help much

 

 

Userlevel 5
Badge +25

Like @davidrich said: without the actual workbench and test data it's hard to give much specific advice.

One thing you can try is have the custom transformer openen when you run the workspace, by looking at the live feature counts you can get an idea of where time is lost

And just so you know, it may be that "an hour" is how long this takes.

Badge +3

looks like the iteration addition, seeming to pose as a index, is not synched with the process.

Is the loop connection not totally superfluous? Looks like you can do this with just one input.

I suspect by the looks of it, that a lot of lists are created not in sequence but simultaneously, which might account for the performance issue.

It is not actually processing "one at a time" afacs (as far as I know...) and can see.

But, you are processing groups. I don't think you need to use a loop at all.

deductive comment based on process structure analysis.

Badge +1

looks like the iteration addition, seeming to pose as a index, is not synched with the process.

Is the loop connection not totally superfluous? Looks like you can do this with just one input.

I suspect by the looks of it, that a lot of lists are created not in sequence but simultaneously, which might account for the performance issue.

It is not actually processing "one at a time" afacs (as far as I know...) and can see.

But, you are processing groups. I don't think you need to use a loop at all.

deductive comment based on process structure analysis.

Hi @gio Yep I am processing groups. The reason I used a loop was because I couldn't work out how to process the attributes per group, without passing the groups individually into the loop...

 

Badge +1

looks like the iteration addition, seeming to pose as a index, is not synched with the process.

Is the loop connection not totally superfluous? Looks like you can do this with just one input.

I suspect by the looks of it, that a lot of lists are created not in sequence but simultaneously, which might account for the performance issue.

It is not actually processing "one at a time" afacs (as far as I know...) and can see.

But, you are processing groups. I don't think you need to use a loop at all.

deductive comment based on process structure analysis.

Oh I understand now! Yes the loop is completely superfluous. I completely over-thought things and I just needed to use a ListBuilder before the PythonCaller and that ensures that each records' group (as defined by the lists I just built) isprocessed individually.

 

The time taken to process the whole file is now only 10.8 seconds!!

 

Thanks so much for pointing this out @gio!

 

Reply