Skip to main content
Released

Async HTTPCaller

Related products:FME Form
paalped
Contributor

I want the HTTPCaller to support async get requests. This could really increase the perfomance of HTTPCaller. Now HTTPCaller wait for every request to come back before it fetches the next request, this is a big waste of resources, the program is basically in idle until it recieves its request.

A naive look into async:

You need to get multiple get requests, so instead of sending a request, getting answer for the request, send a new request, get the answer of the new request and so on. the program would quickly send all requests in a loop without waiting, and then start to recieve requests as the come back in a unordered sequence.

This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

5 replies

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • November 8, 2018

I would like to see this with a parameter controlling the number of simultaneous requests. I don't want to hammer a server, but when it accepts 4-6 simultaneous connections from a single IP address, I would like to take advantage of that fact easily.

 

My current workaround is to put the HTTPCaller in a custom transformer, place a modulo counter just before the CT with the count set to the number of concurrent connections and enable parallel processing with the group by set to the modulo attribute.

J
Forum|alt.badge.img

I am encountering the same query from other FME users in my office. You wouldn't happen to have the custom transformer available to share?

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • March 26, 2019

Unfortunately not, but the only thing in the custom transformer is the HTTPCaller.

paalped
Contributor
Forum|alt.badge.img+5
  • paalpedContributor
  • Author
  • Contributor
  • January 14, 2020

Hi @jasonclemis,

I have now explored the asyncio in FME 2019 With Python 3.7. and its quite easy to do Async get requests With the improved features of 3.7. Are you still interested in this? I can try to bundle it inside a Custom Transfomer.

siennaatsafe
Safer
Forum|alt.badge.img+11

Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail

 

See Concurrency / Looping Options in our HTTPCaller Documentation

Helpful Members This Week

Community Stats

31,140
Posts
118,064
Replies
38,759
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.