Skip to main content
  • EN

Hi I am trying to use FME to update data on AGOL. This works with small data sets, however on larger datasets I get a timeout error (see below), when the process takes more than about 20 seconds.

I am using FME 2015.1.1.0 and can't upgrade at the moment :-(

I have tried using system proxy settings (but this did not make any difference)

If I process the updates in batches this works (but is messy), and I can't truncate a large tables (as this is one large transaction)

Has anyone else had a similar issue with a known work around?

 

FME Error: "The error was 'HTTPSConnectionPool(host='services2.arcgis.com', port=443): Read timed out"

I think we ran into this issue as well with that version. Play around with the "Number of Features to Write" parameter, so there is less data that it tries to write, which should hopefully resolve your issue.

It is a bit more work, but you can use the HTTPCaller to write features to ArcGIS Online. Since they will be written per feature, you want have a timeout. Look at my blog here. Although it focusses on reading, it shows the concept. Combining it with the ArcGIS REST API you can also write features the same way.

I think we ran into this issue as well with that version. Play around with the "Number of Features to Write" parameter, so there is less data that it tries to write, which should hopefully resolve your issue.

 

Thanks runneals I had tried reducing the number to write, which works, but when I try to write to a large dataset the inital trunc process fails (as this takes more than 20 seconds). I am now using ArcMap to load the data, but have found some features are getting dropped, where they don't get dropped if I load via FME. Hopefully I can clean up the data so I can load it all via ArcMap or get around the Trunc process with FME.

It is a bit more work, but you can use the HTTPCaller to write features to ArcGIS Online. Since they will be written per feature, you want have a timeout. Look at my blog here. Although it focusses on reading, it shows the concept. Combining it with the ArcGIS REST API you can also write features the same way.

Thanks lars_de_vries I'll try using the native uploader via ArcGIS for now, but this API approach could be good to learn. It would be appreciated if you could let me know if you end up getting custom transformers to write published.

 

Cheers

 

 

 

Thanks runneals I had tried reducing the number to write, which works, but when I try to write to a large dataset the inital trunc process fails (as this takes more than 20 seconds). I am now using ArcMap to load the data, but have found some features are getting dropped, where they don't get dropped if I load via FME. Hopefully I can clean up the data so I can load it all via ArcMap or get around the Trunc process with FME.
I have a python script that you can use if you are looking to do a truncate prior to writing. Let me know if you'd be interested in that. I would highly suggest you try writing out to a FGDB using the non-api writer, then uploading that file geodatabase to AGOL directly, and use the AGOL writer to truncate/load to that. I know that there are some funky things that esri and FME have when writing to various file types dealing with esri.

Reply


Community Stats

30,092
Posts
114,247
Replies
37,720
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.