Hello,
I am working on writing large amounts of data to ArcGIS portal and due to these data sizes I am getting 502 Server Error: Bad Gateway errors or the token expires before all the data is uploaded.
I have created a workflow that works OK, but isn't perfect, which I'll describe below, but I'm wondering where anyone would have other recommendations.
So the first thing I do is empty a feature service on ArcGIS Portal of any existing data to give an empty data structure to upload new data into. I do this using a Feature Writer with Writer Mode set to DELETE. An alternative would be to use the truncate first option.
Once the deletion has completed I use a Feature Reader to read in all the new data I want to upload. This data's schema matches that of the dataset on the Portal.
To avoid the Bad Gateway and Token errors I divide the data into more manageable volumes and use a workspace runner to batch process the uploading of these into ArcGIS Portal. To do this I add a Counter to give each feature a unique number starting at 0 and then an Attribute Creator using the expression: @Evaluate(@floor(@Value(_count)/200000)) to group features in blocks of 200,000 (typically 2-3 million features to write, but in one case almost 5 million). I use a Feature Writer to write these groups of features out to individual .ffs files with the group number appended to the file name and then use a WorkspaceRunner to kick off a simple workbench that takes each .ffs file and individually uploads groups of 200,000 features into the ArcGIS Portal feature service.
The process runs OK, but I still get a few batches of uploads that return a Bad Gateway error. To fix these I have a separate workbench that compares the data I've loaded via the process above with the data in the .ffs files using a unique ID attribute. Any features where the ID's don't match get uploaded.
I'm wondering if (a) my workflow using the Workspace Runner is the best way to do this batch uploading, (b) whether there's a nice way to capture the Bad Gateway error to retry/restart the Workspace Runner data uploads and (c) what options there are for capturing the logs from the Workspace Runner data upload processes (such as appending a run number to the log file name).
Note I'm using FME Desktop 2019 for this.
Any advice would be gratefully received.
Thanks