Skip to main content
  • EN

Hi - Keen for some thoughts on this.

I'm using an XMLTemplater to prepare a SOAP request and then an HTTPCaller to make the call to a web service. This is working well.

The SOAP request takes a BatchNumber which is default 1.

The service returns data in batches (if there are more than a 1000 records). With the first return a BatchCount is also returned which tells me how many batches there are in total.

With this - If I re-call the service again and set BatchNumber to 2 in the SOAP request - then I'll get the next set and so on.

The problem I have? How do I model this? I have the process running perfectly with 1 batch. Don't know how to effectively re-run the process based on the number of BatchNumbers.

Any thoughts?

Thanks

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

Desktop - Was hoping to avoid a custom transformer but that might be the way.

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

Thanks. Was trying to avoid something custom but will have a look.

Thanks. Was trying to avoid something custom but will have a look.

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Thanks - Managed to get this worked with a looping custom transformer.

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.
You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

Ah! Awesome - Needed another Input. Works well now. Thanks for the assistance.

Reply


Community Stats

30,351
Posts
115,342
Replies
38,083
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.