Skip to main content
Solved

FME to update a large table (non-spatial) in AGOL

  • May 21, 2026
  • 2 replies
  • 69 views
peterhearns
Contributor
Forum|alt.badge.img+8

I have been using FME 2025.1 to update a table that has 1.7M records and 12 fields in ArcGIS Enterprise (AGE) for the last year and we are getting ready to move the webmap to ArcGIS Online (AGOL) later this year.  The issue that I am having is that it is taking more than double the amount of time to update the table in AGOL.  I’ve read the FME community comments and the Esri forums but haven’t been able to find an alternate solution that can improve the update speed.

My typical process has processing/enriching in FME and then the feature writer to truncate and insert into the feature layer in AGOL.  The hosted feature layer also has a “View” as that is the portion that we would eventually show to the public.  The update is run weekly and is automated in AGE using FME Flow, which is also the goal for AGOL.  I am not a python wizard.  

Does anyone have any suggestions or has anyone experienced the same issue and found the solution?

 

Thanks,

Peter

Best answer by donalmateer

Hi ​@peterhearns! Thanks for the post!
Have you had a chance to look at this article? Using Append to Write Big Data to ArcGIS Online in FME

The standard Feature Service writer pushes records in batches via the REST API at 1.7M records that's a huge number of individual HTTP calls. The Append API approach sidesteps this by writing your data to a File Geodatabase or CSV first, then doing a single upload + append call to replace the data. Much more efficient for a weekly full-refresh at this scale.

Let me know if this would work for your case!

    2 replies

    donalmateer
    Safer
    Forum|alt.badge.img+11
    • donalmateerSafer
    • Safer
    • Best Answer
    • May 27, 2026

    Hi ​@peterhearns! Thanks for the post!
    Have you had a chance to look at this article? Using Append to Write Big Data to ArcGIS Online in FME

    The standard Feature Service writer pushes records in batches via the REST API at 1.7M records that's a huge number of individual HTTP calls. The Append API approach sidesteps this by writing your data to a File Geodatabase or CSV first, then doing a single upload + append call to replace the data. Much more efficient for a weekly full-refresh at this scale.

    Let me know if this would work for your case!

      peterhearns
      Contributor
      Forum|alt.badge.img+8
      • peterhearnsContributor
      • Author
      • Contributor
      • May 27, 2026

      Hi ​@peterhearns! Thanks for the post!
      Have you had a chance to look at this article? Using Append to Write Big Data to ArcGIS Online in FME

      The standard Feature Service writer pushes records in batches via the REST API at 1.7M records that's a huge number of individual HTTP calls. The Append API approach sidesteps this by writing your data to a File Geodatabase or CSV first, then doing a single upload + append call to replace the data. Much more efficient for a weekly full-refresh at this scale.

      Let me know if this would work for your case!

      Thanks.  I saw this one and it took a while to figure it out, but it definitely works… and it works fast.  I have an Experience Builder in ArcGIS Enterprise (AGE) and now ArcGIS Online (AGOL).  The full process in AGE takes about an hour to update and when I moved it to AGOL it looked like it would take 4+ hours to update.  Using the REST APIs seems to have reduced the update to 20 minutes.  Now the enrichment part of the data takes the most time rather than the actual truncate and append!  

      Next I am going to try and replicate this in AGE.

        Community Stats

        32,658
        Posts
        123,945
        Replies
        41,071
        Members
        Terms & ConditionsAccessibility statement
        A product of
        Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.