Getting duplicate features when writing to ArcGIS Online

@Mark Stoakes​  thanks for the reply. Will the "Log Debug" option under FME Options --> Translation --> Log Message Filter in version 2021.0.0.0 do the trick or must we be running 2021.1+? We are running ArcGIS Pro 2.8.3 and Interop Extension (FME) 2021.0.0.0 in our production env. We haven't upgraded to Pro 2.9.0 because we're waiting for SP1 (2.9.1). We typically upgrade Pro + Interop (FME) in tandem to ensure compatibility.

@ssharp​ You need FME 2021.1 or higher to get the additional logging. Just a Reminder that you can have multiple versions of FME Desktop installed, just use a different install folder, i.e. c:\\apps\\FME2021.1. You can then continue you run your preferred version of FME for your production jobs.

@Mark Stoakes​ thanks for the quick response. The truncate/append issue is random (might happen 1 day out of 30 daily runs of the ETL), meaning that running it once or twice in a "test" env is not likely to trap the issue. We would likely need to create a seperate ETL for testing, having it run on a loop until the number of records in the "read source" (SQL table) failed to match the number of records in the "write target" (AGO table). My hope was/is that Esri/Safe's devops team would do that, but maybe not?

@bruceharold​ make note of Marks comment regarding the "need" to be running FME 2021.1 for the "enhanced" logging to work. Any idea when SP1 for Pro 2.9.0 will be released?

SP1 is expected January 11th 2022.

I have been experiencing this issue consistently for the last couple of months since my organization started using FME for large daily data overwrites (200,000+ records) , but in the last two work days, I have finally not received any duplicate records. Is anyone else experiencing this, or have I just experienced a couple of extremely lucky days? I am working in FME 2021.2 and sending data to ArcGIS Online.

@dwaltz​ I suspect this is good luck. We're pretty sure that the problem occurs when a time out or network reconnect occurs. FME 2021.1 and higher has improved logging for AGOL reading, so if this does start again: Turn on the debug logging under the Tools menu -> FME Options -> Translation -> Log Message disclosure panel and check Log Debug. If you can send us a log file from a job that results in duplicates that might help us get this resolved. You can post the log file here or send it directly to me mark @ safe.com

Was this issue solved? I am having the same problem. I will try out the workaround suggested by @bruceharold​ in the meantime. But I just wanted to know if there was a different solution.

Hi @gis_mapper​, unfortunately, this issue remains. I've attached a Workspace that handles the truncate-append flow discussed in Bruce's article.

We have not been able to reproduce this reliably in order to fix it. Logging in more recent versions of FME Server has been improved, so if you have an FME Server job log file, preferably with Log Debug mode on, that you could share that would be greatly appreciated! If you aren't comfortable sharing it with the Community, please submit a case to share it with us.

Also, for anyone experiencing this issue randomly, you can avoid duplicates by supplying your own OBJECTID/GLOBALID when inserting features, instead of letting AGOL assign them. That way, a retried write won’t create duplicate rows. You will receive an error instead, but at least it will prevent duplicates.

FME 2023 will introduce an asynchronous writes option for the ArcGIS Online Feature Service Writer, which is unlikely to have the same issues with timeouts and retries.

That is a really good tip!

Hi @mattmatsafe​ , I have already tried this option and I get a message that the translation failed. So sadly, this is not a solution for me. Or maybe I am adding the wrong parameters. I selected yes for both Use Global IDs and the Preserve Global ID options. Is there an additional option I need to add to make this work? I am trying the change detector but until now it has only deleted my AGOL data but it is not inserting. I selected the Upsert but I get a message saying it is not supported. I will continue looking for a solution. Thanks.

@gis_mapper​ Does the translation fail now because of duplicate IDs? Please feel free to submit a case with your failed job log and we can take a closer look. If you can also share a job log that shows the timeout that leads to duplicates, with Log Debug mode on (turned on in Workbench under FME Options > Translation > Log Message Filter), then we can take a closer look at what is going on and hopefully determine the root cause.

Hello @mattmatsafe​ I submitted a ticket and I included the complete log using a change detector. Thanks.

Hi @gis_mapper​ , thanks I see you created a case the other day with the desktop team and that Dan has been working with you. I don't see any timeout or retry in log file provided though, so yours may be a different issue. All HTTP responses from AGOL are coming back 200 successful. If you have a log file where you experienced the timeout, please add it to the case - I'll let Dan know. Thanks!