The HTTPCaller is a great transformer for making HTTP requests, but I have quite a few requests is there a way to speed up this process?
Solved
I am making lots of HTTP requests using the HTTPCaller, is there a way to speed this up?
+18- Safer
Best answer by siennaatsafe
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
+11- Safer
- Best Answer
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
+17- Contributor
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future.
https://storage.googleapis.com/openimages/web/index.html
+17- Contributor
siennaatsafe wrote:
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
+5- Contributor
siennaatsafe wrote:
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
This made my day :)
+8- Contributor
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
jeroen wrote:
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, thanks for your question. I would encourage you to post this as a new idea on our Ideas page, and share with us what your intended use case would look like. This information would also be very helpful for us. Thank you!
+11- Safer
jeroen wrote:
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen,
It is something that is on our radar. We don't have any plans for the immediate future as it will be quite a big change. Stay tuned to this idea, if any updates do occur, I'll be sure to post here https://knowledge.safe.com/content/idea/34725/httpcaller-to-support-parallel-processing.html Also, adding your use case is always useful!
+5- Contributor
jeroen wrote:
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, its quite simple to make this run on linux. I just need to make it pass the testsuit on fme hub and it will be available.
+24- Supporter
bruceharold wrote:
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
+17- Contributor
sigtill wrote:
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
If he isn't I will be....
+44- Safer
bruceharold wrote:
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
+17- Contributor
mark2atsafe wrote:
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
Brockton Oval is near the Vancouver convention centre, organise a game during your user conference!
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.