Skip to main content
Question

AGOL Writing - Limited to ~20 seconds

N
Forum|alt.badge.img

Hi I am trying to use FME to update data on AGOL. This works with small data sets, however on larger datasets I get a timeout error (see below), when the process takes more than about 20 seconds.

I am using FME 2015.1.1.0 and can't upgrade at the moment :-(

I have tried using system proxy settings (but this did not make any difference)

If I process the updates in batches this works (but is messy), and I can't truncate a large tables (as this is one large transaction)

Has anyone else had a similar issue with a known work around?

 

FME Error: "The error was 'HTTPSConnectionPool(host='services2.arcgis.com', port=443): Read timed out"

5 replies

runneals
Contributor
Forum|alt.badge.img+22
  • runnealsContributor
  • Contributor
  • May 26, 2017

I think we ran into this issue as well with that version. Play around with the "Number of Features to Write" parameter, so there is less data that it tries to write, which should hopefully resolve your issue.

lars_de_vries
Forum|alt.badge.img+10

It is a bit more work, but you can use the HTTPCaller to write features to ArcGIS Online. Since they will be written per feature, you want have a timeout. Look at my blog here. Although it focusses on reading, it shows the concept. Combining it with the ArcGIS REST API you can also write features the same way.

N
Forum|alt.badge.img
runneals wrote:

I think we ran into this issue as well with that version. Play around with the "Number of Features to Write" parameter, so there is less data that it tries to write, which should hopefully resolve your issue.

 

Thanks runneals I had tried reducing the number to write, which works, but when I try to write to a large dataset the inital trunc process fails (as this takes more than 20 seconds). I am now using ArcMap to load the data, but have found some features are getting dropped, where they don't get dropped if I load via FME. Hopefully I can clean up the data so I can load it all via ArcMap or get around the Trunc process with FME.
N
Forum|alt.badge.img
lars_de_vries wrote:

It is a bit more work, but you can use the HTTPCaller to write features to ArcGIS Online. Since they will be written per feature, you want have a timeout. Look at my blog here. Although it focusses on reading, it shows the concept. Combining it with the ArcGIS REST API you can also write features the same way.

Thanks lars_de_vries I'll try using the native uploader via ArcGIS for now, but this API approach could be good to learn. It would be appreciated if you could let me know if you end up getting custom transformers to write published.

 

Cheers

 

 

runneals
Contributor
Forum|alt.badge.img+22
  • runnealsContributor
  • Contributor
  • June 1, 2017
nichowardwork wrote:

 

Thanks runneals I had tried reducing the number to write, which works, but when I try to write to a large dataset the inital trunc process fails (as this takes more than 20 seconds). I am now using ArcMap to load the data, but have found some features are getting dropped, where they don't get dropped if I load via FME. Hopefully I can clean up the data so I can load it all via ArcMap or get around the Trunc process with FME.
I have a python script that you can use if you are looking to do a truncate prior to writing. Let me know if you'd be interested in that. I would highly suggest you try writing out to a FGDB using the non-api writer, then uploading that file geodatabase to AGOL directly, and use the AGOL writer to truncate/load to that. I know that there are some funky things that esri and FME have when writing to various file types dealing with esri.

Reply


Helpful Members This Week

Community Stats

31,478
Posts
119,339
Replies
39,093
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.