Writing LARGE datasets to ArcGIS Online

Very cool David. Also check it out @1spatialdave

@carsonlam Hi, this question could be refactored as an idea, namely to support bulk appends when the arcgis Python API is available, with and without truncate. Personally I use Pro to populate feature services and use DI/FME for maintenance operations but others may like a bulk writer.

Is there a similar bulk append mechanism in the ArcGIS REST API?

https://developers.arcgis.com/rest/services-reference/truncate-feature-layer-.htm

https://developers.arcgis.com/rest/services-reference/append-feature-service-.htm

Hi @bruceharold,

Unfortunately we cannot convert questions to ideas, but I have created the idea for you to vote on. Please see Support bulk appends when the ArcGIS Python API becomes available

- Liz

cc: @runneals, @mmccart, @1spatialdave, @carsonlam

Thanks Liz! Maybe Carson will make an end run around the idea and do it with the REST API so there is no ArcGIS dependency. I'm seeing a lot of action with bulk work to AGOL so it will be popular.

Great minds think alike @carsonlam :) I was going to go that way with the next enhancements for improving the script, as using the ArcGIS Pro API requires quite a bit of overhead and increases the run time quite a bit. However my workload exceeds my free time, so probably won't be happening for a while on my end.

If you need a use case, we have a workspace that runs every week that updates historic rail lines in Iowa that takes almost 30 minutes to run, and almost 20 minutes of that is just writing data up to ArcGIS online, so using this bulk append method would only take a couple minutes and speed it up quite a bit.

I was successfully able to get over the major issue when the file geodatabase would get stuck on ArcGIS Online due to the delete portion of the script not being executed. See the sample below for the best implementation that works now (using feature writer and a python caller)!

Sample_Workspace_AGOL_Append_PUBLIC.fmw