Skip to main content
Solved

HTTPCaller Dealing With Batches

  • April 16, 2019
  • 11 replies
  • 77 views
jiriteach
Forum|alt.badge.img

Hi - Keen for some thoughts on this.

I'm using an XMLTemplater to prepare a SOAP request and then an HTTPCaller to make the call to a web service. This is working well.

The SOAP request takes a BatchNumber which is default 1.

The service returns data in batches (if there are more than a 1000 records). With the first return a BatchCount is also returned which tells me how many batches there are in total.

With this - If I re-call the service again and set BatchNumber to 2 in the SOAP request - then I'll get the next set and so on.

The problem I have? How do I model this? I have the process running perfectly with 1 batch. Don't know how to effectively re-run the process based on the number of BatchNumbers.

Any thoughts?

Thanks

Best answer by david_r

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

david_r
Celebrity
  • david_rCelebrity
  • Best Answer
  • April 16, 2019

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

nielsgerrits
VIP
Forum|alt.badge.img+61

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

Buckle your seatbelt Dorothy, cause Kansas is going bye-bye...
jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 16, 2019

Desktop or Server?

If Desktop, you could build a custom transformer and loop through all the batch numbers. Or use a WorkspaceRunner to fire a second workbench for each batch number.

Personally I prefer WorkspaceRunners over custom transformers. Let the parent handle the main part, and create a databasetable with jobs. Let one or more children process these jobs and update the status in the table. This way the work is split in doable portions. If one portion fails, the job is not updated and only the failed job can be restarted. Instead of the whole, time consuming, proces. But it all depends on the difficulty / volumes / timeouts etc.

If Server, you could use a FMEServerJobSubmitter instead of a WorkspaceRunner.

Desktop - Was hoping to avoid a custom transformer but that might be the way.

jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 16, 2019

You can use a looping custom transformer:

http://docs.safe.com/fme/2019.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

 

Thanks. Was trying to avoid something custom but will have a look.

david_r
Celebrity
  • david_rCelebrity
  • April 16, 2019

Thanks. Was trying to avoid something custom but will have a look.

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 16, 2019

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Thanks - Managed to get this worked with a looping custom transformer.

jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 16, 2019

It's basically the only way to loop in FME within a single workspace (unless you count doing it in e.g. Python).

The WorkspaceRunner suggested by @nielsgerrits is a good alternative.

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

david_r
Celebrity
  • david_rCelebrity
  • April 16, 2019

Running into some issues - Any ideas on how to increment my running BatchNumber after each run? What can I use to do this and at the same time check if the BatchNumber < BatchCount then don't loop. Thanks

You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.
jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 17, 2019
You can use the ExpressionEvaluator to increment the BatchNumber by 1. Then check with a Tester if BatchNumber > BatchCount to stop looping.

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

david_r
Celebrity
  • david_rCelebrity
  • April 17, 2019

Thanks - I have tried to use an ExpressionEvaluator but perhaps I have the syntax wrong. Just @Value(_batchNumber)+1 ? Problem is when it loops - _batchNumber isn't being passed as 2 for example. Any ideas? Thanks

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

jiriteach
Forum|alt.badge.img
  • jiriteach
  • Author
  • April 17, 2019

Just make sure your new value for _batchNumber isn't being overwritten when it loops back to the start. You can e.g. use a second Input that you loop back into:

Ah! Awesome - Needed another Input. Works well now. Thanks for the assistance.

Community Stats

32,129
Posts
121,967
Replies
39,779
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.