No experience with the AsyncHTTPCaller but in FME 2021 the HTTPCaller can do up to 100 transfers at a time out of the box. See this Idea and the documentation. (Look for Concurrency / Looping Options.)

This parameter controls how features are processed through the transformer. The value of this parameter indicates the maximum number of transfers that will be in progress at the same time.

When the parameter is set to 1, a transfer will be fully processed before the next transfer can begin.

When the parameter is set to a value greater than 1, the transformer will not wait for each transfer to finish before starting the next one. There may be multiple transfers occurring at the same time. This can provide significant performance benefits. This value should be adjusted based on the ability of the HTTP server to handle multiple requests at the same time, taking into consideration that multiple HTTPCaller transformers may be accessing the server at the same time.

This parameter must be set to 1 if the transformer is being used inside a custom transformer which has a loop output.

Thanks!

The website I take the data from allows for 100 requests per second. Could you help me out how to implement this in the new HTTP Caller?

My goal is to download as soon as possible without exceeding this limit.

Hi @lambertus​ it is possible to set a limit on the number of HTTP requests the transformer will start in a particular time interval using Rate Limiting. In the screenshot attached you can see I'm able to limit the HTTP Caller to 100 requests per second. Hope this helps!

Thanks! What is the relation between the Rate Limiting parameter and the Concurrency/Looping parameter? Is the behaviour of the latter limited by the Rate Limiting parameter values? Hence, sending requests without waiting before the requests sent before did return, with the limitation of 100 requests/second.

@lambertus​ the Concurrency/Looping parameter is used to denote if you want to send multiple concurrent requests at a time to the HTTP server. This value should also be adjusted based on the ability of the HTTP server to meet such demands. When Rate Limiting is enabled, concurrent requests may still be sent according to the Maximum Number of Concurrent HTTP Requests parameter, but no more than the Maximum Requests per Interval will be sent to the HTTP Server. If you find the HTTP Caller is not meeting your needs and is no where near meeting the 100 requests per second, you could try increasing the Maximum Number of Concurrent HTTP Requests to speed up your workspace.

In any case, the Rate Limiting parameters will help prevent you from over-requesting. You can learn more about this in the HTTP Caller documentation.