Solved

HTTPCaller Dealing With Batches

jiriteach
Badge

Hi - Keen for some thoughts on this.

I'm using an XMLTemplater to prepare a SOAP request and then an HTTPCaller to make the call to a web service. This is working well.

The SOAP request takes a BatchNumber which is default 1.

The service returns data in batches (if there are more than a 1000 records). With the first return a BatchCount is also returned which tells me how many batches there are in total.

With this - If I re-call the service again and set BatchNumber to 2 in the SOAP request - then I'll get the next set and so on.

The problem I have? How do I model this? I have the process running perfectly with 1 batch. Don't know how to effectively re-run the process based on the number of BatchNumbers.

Any thoughts?

Thanks

icon

Best answer by david_r 16 April 2019, 11:31

View original

11 replies

david_r
Rank
Userlevel 4

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

nielsgerrits
Rank
Userlevel 6
Badge +32

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

jiriteach
Badge

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

Desktop - Was hoping to avoid a custom transformer but that might be the way.

jiriteach
Badge

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

Thanks. Was trying to avoid something custom but will have a look.

david_r
Rank
Userlevel 4

Thanks. Was trying to avoid something custom but will have a look.

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

jiriteach
Badge

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Thanks - Managed to get this worked with a looping custom transformer.

jiriteach
Badge

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

david_r
Rank
Userlevel 4

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.
jiriteach
Badge
You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

david_r
Rank
Userlevel 4

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

jiriteach
Badge

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

Ah! Awesome - Needed another Input. Works well now. Thanks for the assistance.

Reply


Community Stats

28,666
Posts
109,339
Replies
36,535
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.