Replace JobWaiter with HTTPCaller

I tried longPoll parameter TRUE and received a 200 for the submitted jobs. Although the response body has no status information.

And the Jobs submitting workspace ended before the submitted jobs. So not really a progress. 

Firstly, I suspect/hope Safe will create a replacement transformer for this tool. So hopefully you wont have to create it yourself.

Secondly, why are you not setting the wait until complete to Yes if you’re also using a Job Waiter anyway?

With respect to your question - you’re on the right track with the REST call.

Because version 3 or f the rest api is being deprecated, however, you should instead be using version 4 of the REST api. You should find the API doc at:
 

HOST/fmeapiv4/docs/index.html

The call you want to use is:
 

GET /fmeapiv4/jobs/{jobid}

In this case the {jobid} should be returned from the FMEServerJobSubmitter. 

You will then need to parse the _reponse_body attribute, you can use a JSONFlattener. The “status” is what you’re looking for here. 

Now comes the tricky part….You need to create a looping transformer to repeat the call until you get a final status.
 

That’s exactly what I try to avoid. FME is still cumbersome if it comes to looping. Http calls with looping would rather end up in a PythonCaller than using the FME transformers.

So I hope (as you do) safe wont leave me/us alone with the JobWaiter deprecation and will provide a replacement for it instead.

thx for feedback

Klaus

I am also very curious to the answer from Safe on this issue, I do hope that Safe will provide a workaround or another solution and wont let us ‘hanging’….also when is the API V4 going to be released? Will Safe let us know in advance? or wil we notice it once our processing is broken?😟

Hi @itay and @kalbert,

We currently have no plans to replace the JobWaiter.

Have you explored using the Split/Merge block in Automations to accomplish this? Is there any specific functionality that you feel is missing for your use case?

Also, regarding V3, we are still actively working on V4, so there's no announcement yet. For V3 deprecation, I recommend keeping an eye on the FME Deprecations page for any updates: FME Deprecations Page.

Looking forward to hearing more!

Best regards,
Sienna

Hi,

No I have not was hoping that the transformer or a workaround will be made available, can you explain how that can be accomplished with the items you mention.

Hi @itay, 

Thank you for the feedback. I’ve passed it onto our product managers. It’s good to know when deprecated functionality will be missed by users. 

From the product management perspective, the transformer had low usage and we believe Automations offered a better and more efficient workaround than the transformer. However, please let us know if Automations don’t meet your needs. The product development team was curious to know more about your use case.

The workaround in automation might look something like this. This workflow is initiated by a manual trigger, however it could be any other trigger like a schedule or a directory watch. 

From there, 5 workspaces will be initiated to run in parallel. If all the workspace are successful a message will be sent to the checkmark port of the split merge block. If a single action fails, a message will be sent to the “X” port. The best thing about this method is it should be more efficient and won’t require a poll on the Flow instance which caused users issues in the past. 

I’m happy to chat in depth more about your use case and the transition to automations. Just send me a direct message and we can set up a time to discuss. 

Thank you,

Sienna
 

Hi,

I have sent you a direct message to continue this discussion.

Hi @siennaatsafe 

Automation might be a good solution if you start a new ETL project from scratch. For customers running larger ETL programs orchestrated by let’s name it master or controller workspaces doing complex work it’s simply too expensive migrating everything to Automation. 

However we are going to replace the JobWaiter with a custom looped HttpCaller. Reason is that we’ve recognized some unexpected latency of the JobWaiter. We see jobs finished but up to 1h latency before the JobWaiter reacts. Replaced with a looped HttpCaller it’s just a second. (FME Flow 2024.1.)

kind regards

Klaus

Hi ​@siennaatsafe,

For what it's worth, I've run into the same issue - we have a master workspace that calls 26 other workspaces and waits until they are all finished before moving on. Now that we are upgrading to FME 2024, I got the warning messages. For the sake of backward-compatibility, I'd much rather see a replacement transformer to JobWaiter than having to replace it with Automations.

My question now is, can we keep the JobWaiter until the next upgrade (likely in 2026 or 2027)? From your point of view, is there any urgency in replacing it?

Thank you,

Vini

Hi ​@vmtng, 
 

I’ve passed your feedback onto the product management team. I suspect the decision to not support these transformers in the future is firm. However, I’ll be sure to update this post if something does change. 

As long as you are on 2024 there is no need to replace your workflows. However, you’ll likely need to before your next upgrade. Please keep an eye on FME Deprecations Page for the official deprecation. I don’t believe we have an official timeline yet.

Thank you, Sienna!

​@siennaatsafe 
We use FMEFlowJobWaiter in many strategic scheduled automated workflows already to enable significant parallel processing jobsubmitter workflows that load large SDE tables using multiple server engines.  Essentially workflow is:

1. Drop spatial index
2. Create features for different whereclauses to use to divide the sde table to be loaded in multiple chunks in parallel ( mod(objectid, 10)=<featureattrib>) if nothing else suffices)
3. Pass to JobSubmitter with no waiting, to call workspace that loads table(s) dynamically, using the whereclauses provided to do differerent subsets in parallel.
4. FeatureHolder to let all jobs get submitted
5. FMEServerJobWaiter / FMEFlowJobWaiter to wait for jobs to finish in all 4 regional fme servers.
6. Aggregator, then carry on with processing workflow.

This is especially useful when running identical jobs to build georedundant data in multiple time zones because of the inability to join engines to an fme server in another time zone, so we have to set up a whole additional fmeserver in that time zone and submit jobs to it.

Furthermore, while some of us can handle splitting things up to do the necessary rest api stuff using httpcaller, it’s not something I’d trust just anyone to learn and get right without a lot of fumbling if they’ve not done REST requests manually before.  The deprecation details for this say to go see how to use the REST API to replace it, based on the FME Community posts.  This is the only one I found, and it doesn’t have a sample workspace nor clear steps.  The JobWaiter made this simple and easy, just like everything we love about FME.

Please bring it back. 

P.S. The FMEFlowLogFileRetriever was nice for pulling logs from the other time zone FME servers too for listing errors when jobs failed too… that’s probably less commonly used, but it was useful for us.

Hi ​@lanthar,

Unfortunately, we won’t be bringing back those transformers—at least not in the short term. However, we are working on providing articles and examples to help configure the HTTPCaller, so that even newer FME users can successfully transition away from the older transformers.

That said, I completely understand that making this change may require significant time and resources. If you’re willing, it would be incredibly helpful if you could create a case or contact me directly. This would allow us to better understand your specific scenario and ensure we’re supporting you effectively.

Ideally, all the functionality you relied on from those transformers can be accomplished through Automations. If not, I’d love to work with you to identify enhancements that could close the gap. Automations offer significant benefits over JobWaiters and JobSubmitters—including more transparent, readable workflows, improved logging, and more reliable retry mechanisms.

In short, we want to help you over this hurdle and make sure FME continues to support your workflows better than ever before.