Skip to main content
  • EN

We have a workspace in our cloud that pulls data from our own api.

We do one get call per table/api endpoint in parallel, and have about 10 httpcallers that run at once.

We are finding that when we reach about 750mb of data in transfer, we get failures. The last table before failing looks like it has a very small amount of data, when in reality it has a significant amount more (see Containers in the logfile).

Is there a limit on transfer size in FME? We figured we could try offsetting each call but it's strange that it just fails. I can provide more details but thought I'd start here.

 

Message is "An error occurred while accessing the url"...

Note, pulling any table individually through the api is perfectly fine...Pulling all at once through the API causes this error when we reach about 770mb + of data being transferred....

 

Hi @Candace Low​, there's no transfer size limit with the HTTPCaller, however the HTTP 500 error points to a server error so you can also see if there happens to be a limit with the server you're accessing the data from.

You could also try to lower the number of transfers happening in parallel in your workspace. I hope this helps!

Reply


Community Stats

30,162
Posts
114,483
Replies
37,809
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.