The HTTPCaller is a great transformer for making HTTP requests, but I have quite a few requests is there a way to speed up this process?
Best Answer
I am making lots of HTTP requests using the HTTPCaller, is there a way to speed this up?
+21- Safer
Best answer by siennaatsafe
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+12- Safer
- Best Answer
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
+19- Supporter
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future.
https://storage.googleapis.com/openimages/web/index.html
+19- Supporter
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
+5- Contributor
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
This made my day :)
+15- Contributor
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, thanks for your question. I would encourage you to post this as a new idea on our Ideas page, and share with us what your intended use case would look like. This information would also be very helpful for us. Thank you!
+12- Safer
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen,
It is something that is on our radar. We don't have any plans for the immediate future as it will be quite a big change. Stay tuned to this idea, if any updates do occur, I'll be sure to post here https://knowledge.safe.com/content/idea/34725/httpcaller-to-support-parallel-processing.html Also, adding your use case is always useful!
+5- Contributor
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, its quite simple to make this run on linux. I just need to make it pass the testsuit on fme hub and it will be available.
+25- Supporter
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
+19- Supporter
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
If he isn't I will be....
+59- Safer
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
+19- Supporter
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
Brockton Oval is near the Vancouver convention centre, organise a game during your user conference!