Assuming one is a responsible netizen then HTTPCaller should be able to handle multiple concurrent calls and send & catch interleaved jobs. I want to hit the REST endpoint of an ArcGIS geoprocessing service and I know how many service instances there are and I know I'm the only user so I want to have at it.
Released
HTTPCaller to support parallel processing
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+25- Supporter
I`m sure you know about the workaround, putting the HTTPCaller within a Custom Transformer and using parallell processing on that - however, I agree - that would be a usefull transformer to have support for parallell processing built in.
+4Can you elaborate on this or send a screenshot of a workspace? I'm trying to create share links on 18,000+ documents in Box and it's mighty slow just passing each record through a single http caller? Setting up httpcaller in some sort of parallel processing sounds interesting.
Thanks
James
Definitely a good idea. The internet and API's are inherenently multi-threaded/multi-connection based. I'm amazed that FME cannot support them currently, especially given Safe's push in the last few years towards web-services. This seems like a significant oversight.
+22- Safer
Good points all. The FME 2017 Javascript caller supports node.js and as such has a form of multi-processing in it. But having it baked into the HTTPCaller is a very interesting idea for certain.
+11- Contributor
@sigtill
I have this requirement. Can you pls send me the sample fmw file which is calling javasciptcaller node.js mode and httpcaller inside custom transformer
+5- Contributor
Not quite, HTTPCaller should by default be async, with nonblocking sockets. no need for parallelism.
The HTTPCaller may be async and using nonblocking sockets in its implementation (see custom transformer workaround) but using the transformer right of the box makes the whole workspace wait for a response before continuing to the next feature!
Being able to set parameters for parallel processing (of features) would really help! Even if you are a irresponsible netizen! FME being the first choice for DDosing WMS/WFS/geoprocessing services should be Safe criteria for a good HTTPCaller transformer ;-)
+19- Influencer
Parallel processing is process-based and really expensive, I can't get any gains from it. HTTP requests are super lightweight so hopefully easier to parallelize.
+5- Contributor
Hi, I did some work on a AsyncHTTPCaller, Im not avle to make it pass the test on hub, (because the hub runs fme server with python 3.6, and dont permit installing packages). But if the need is still there I put it on github: https://github.com/paalped/AsyncHTTPCaller
+19- Influencer
Hi Paal, I tried your transformer, it is brilliant! @siennaatsafe see Paal has put his project on GitHub.
+13- Supporter
Clear example about using a custom trasnformer for parallell processing https://knowledge.safe.com/articles/86723/parallel-processing-using-custom-transformers.html
+12- Safer
I have some great news! This feature is currently in the 2021.0 betas on https://www.safe.com/support/downloads/#next-beta
Now in the HTTPCaller there is a new parameter, “Process Multiple Transfers in Parallel“
This parameter controls how features are processed through the transformer. When “Yes” is selected, multiple HTTP transfers can be in progress at the same time. That is, the transformer will not wait for one transfer to finish before starting the next one. Up to 100 transfers can be in progress at the same time, which can provide substantial performance benefits. When “No” is selected, each transfer will be processed to completion before the next transfer is started.
In my initial test of 50 features, before, it took 13 seconds and now it takes 1 second.
Login to the community
No account yet? Create an account
An FME Account is required to contribute
LoginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.