Question

Is there a transfer size limit with multiple httpcallers in a workspace?

  • 17 May 2023
  • 1 reply
  • 1 view
candacel

We have a workspace in our cloud that pulls data from our own api.

We do one get call per table/api endpoint in parallel, and have about 10 httpcallers that run at once.

We are finding that when we reach about 750mb of data in transfer, we get failures. The last table before failing looks like it has a very small amount of data, when in reality it has a significant amount more (see Containers in the logfile).

Is there a limit on transfer size in FME? We figured we could try offsetting each call but it's strange that it just fails. I can provide more details but thought I'd start here.

 

Message is "An error occurred while accessing the url"...

Note, pulling any table individually through the api is perfectly fine...Pulling all at once through the API causes this error when we reach about 770mb + of data being transferred....

 

1 Attachment

1 reply

saraatsafe
Rank
Userlevel 1
Badge +6

Hi @Candace Low​, there's no transfer size limit with the HTTPCaller, however the HTTP 500 error points to a server error so you can also see if there happens to be a limit with the server you're accessing the data from.

You could also try to lower the number of transfers happening in parallel in your workspace. I hope this helps!

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.