The HTTPCaller is a great transformer for making HTTP requests, but I have quite a few requests is there a way to speed up this process?
Best answer by siennaatsafe
View original
I am making lots of HTTP requests using the HTTPCaller, is there a way to speed this up?
+9
- Contributor
- 176 replies
-
5 February 2020
- Best Answer
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
+16
- Contributor
- 325 replies
-
5 February 2020
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future.
https://storage.googleapis.com/openimages/web/index.html
+16
- Contributor
- 325 replies
-
5 February 2020
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
+4
One of our users @paalped recently created a custom transformer on the FME Hub to make asynchronous requests using the HTTPCaller. It's called the AsyncHTTPCaller and is available to download in workbench. It currently only supports GET Requests and is available in FME 2019.2+ on Windows. From my testing, getting 100 images from a website, the AsyncHTTPCaller was around 9x faster than the HTTPCaller. For more information see the custom transformer page https://hub.safe.com/publishers/paalped/transformers/asynchttpcaller
Update, May 18th, 2021! This functionality has now been released in FME 2021.0 https://community.safe.com/s/idea/0874Q000000TkurQAC/detail
See Concurrency / Looping Options in our HTTPCaller Documentation
This made my day :)
+7
- Contributor
- 93 replies
-
10 February 2020
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, thanks for your question. I would encourage you to post this as a new idea on our Ideas page, and share with us what your intended use case would look like. This information would also be very helpful for us. Thank you!
+9
- Contributor
- 176 replies
-
14 February 2020
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen,
It is something that is on our radar. We don't have any plans for the immediate future as it will be quite a big change. Stay tuned to this idea, if any updates do occur, I'll be sure to post here https://knowledge.safe.com/content/idea/34725/httpcaller-to-support-parallel-processing.html Also, adding your use case is always useful!
+4
Any plans for Safe to make this transformer available for Linux as well? I know the CT from the hub is not made by Safe but it would be really nice to have this transformer available for FME Cloud/Server.
Hi @jeroen, its quite simple to make this run on linux. I just need to make it pass the testsuit on fme hub and it will be available.
+21
- Contributor
- 955 replies
-
14 February 2020
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
+16
- Contributor
- 325 replies
-
14 February 2020
Ajaj! @dmitribagh @dmitri_bagh will be busy with this webpage from now!
If he isn't I will be....
Userlevel 4
+25
- Safer
- 2399 replies
-
14 February 2020
@siennaatsafe I believe the site you hit contained images suitable for AI model training, I predict a dynamite blog from @mark2atsafe in our future!
https://storage.googleapis.com/openimages/web/index.html
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
+16
- Contributor
- 325 replies
-
14 February 2020
I thought, yes, this is definitely more in Dmitri's wheelhouse. But then I saw "cricket ball" is a defined object in their classes. Now I have to give it a shot!
Brockton Oval is near the Vancouver convention centre, organise a game during your user conference!
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.