When writing to a hosted feature layer in ArcGIS Online we are getting duplicate features on occasion. In the latest run where this happened, there were 15,304 features in the dataset which were reported as written to ArcGIS Online, although the feature layer actually had 16,304 features written to it. Features with OBJECTID 9001 through 10000 were exactly duplicated making it seem like a batch of 1,000 features were written exactly twice during the process. OBJECTID 10001 was an exact duplicate of 9001, 10002 was a dupe of 9002 and so on.... After this batch of duplicates, subsequent features were not duplicated (features before OBJECTID 9001 were also not duplicated).

Getting duplicate features when writing to ArcGIS Online

@jasonschroeder @bruceharold @ssharp @dnfox

FME 2021.1 and higher has improved logging that might help us trace the cause of this issue. Turn on the debug logging under the Tools menu -> FME Options -> Translation -> Log Message disclosure panel and check Log Debug. If you can send us a log file from a job that results in duplicates that might help us get this resolved. You can post the log file here or send it directly to me mark @ safe.com

@Mark Stoakes thanks for the reply. Will the "Log Debug" option under FME Options --> Translation --> Log Message Filter in version 2021.0.0.0 do the trick or must we be running 2021.1+? We are running ArcGIS Pro 2.8.3 and Interop Extension (FME) 2021.0.0.0 in our production env. We haven't upgraded to Pro 2.9.0 because we're waiting for SP1 (2.9.1). We typically upgrade Pro + Interop (FME) in tandem to ensure compatibility.

M

+2

@Mark Stoakes thanks for the reply. Will the "Log Debug" option under FME Options --> Translation --> Log Message Filter in version 2021.0.0.0 do the trick or must we be running 2021.1+? We are running ArcGIS Pro 2.8.3 and Interop Extension (FME) 2021.0.0.0 in our production env. We haven't upgraded to Pro 2.9.0 because we're waiting for SP1 (2.9.1). We typically upgrade Pro + Interop (FME) in tandem to ensure compatibility.

@ssharp You need FME 2021.1 or higher to get the additional logging. Just a Reminder that you can have multiple versions of FME Desktop installed, just use a different install folder, i.e. c:\\apps\\FME2021.1. You can then continue you run your preferred version of FME for your production jobs.

@Mark Stoakes thanks for the reply. Will the "Log Debug" option under FME Options --> Translation --> Log Message Filter in version 2021.0.0.0 do the trick or must we be running 2021.1+? We are running ArcGIS Pro 2.8.3 and Interop Extension (FME) 2021.0.0.0 in our production env. We haven't upgraded to Pro 2.9.0 because we're waiting for SP1 (2.9.1). We typically upgrade Pro + Interop (FME) in tandem to ensure compatibility.

@Mark Stoakes thanks for the quick response. The truncate/append issue is random (might happen 1 day out of 30 daily runs of the ETL), meaning that running it once or twice in a "test" env is not likely to trap the issue. We would likely need to create a seperate ETL for testing, having it run on a loop until the number of records in the "read source" (SQL table) failed to match the number of records in the "write target" (AGO table). My hope was/is that Esri/Safe's devops team would do that, but maybe not?

@bruceharold make note of Marks comment regarding the "need" to be running FME 2021.1 for the "enhanced" logging to work. Any idea when SP1 for Pro 2.9.0 will be released?

+16

bruceharold
Contributor
326 replies
2 years ago
29 November 2021

@Mark Stoakes thanks for the reply. Will the "Log Debug" option under FME Options --> Translation --> Log Message Filter in version 2021.0.0.0 do the trick or must we be running 2021.1+? We are running ArcGIS Pro 2.8.3 and Interop Extension (FME) 2021.0.0.0 in our production env. We haven't upgraded to Pro 2.9.0 because we're waiting for SP1 (2.9.1). We typically upgrade Pro + Interop (FME) in tandem to ensure compatibility.

SP1 is expected January 11th 2022.

D

I have been experiencing this issue consistently for the last couple of months since my organization started using FME for large daily data overwrites (200,000+ records) , but in the last two work days, I have finally not received any duplicate records. Is anyone else experiencing this, or have I just experienced a couple of extremely lucky days? I am working in FME 2021.2 and sending data to ArcGIS Online.

M

+2

I have been experiencing this issue consistently for the last couple of months since my organization started using FME for large daily data overwrites (200,000+ records) , but in the last two work days, I have finally not received any duplicate records. Is anyone else experiencing this, or have I just experienced a couple of extremely lucky days? I am working in FME 2021.2 and sending data to ArcGIS Online.

@dwaltz I suspect this is good luck. We're pretty sure that the problem occurs when a time out or network reconnect occurs. FME 2021.1 and higher has improved logging for AGOL reading, so if this does start again: Turn on the debug logging under the Tools menu -> FME Options -> Translation -> Log Message disclosure panel and check Log Debug. If you can send us a log file from a job that results in duplicates that might help us get this resolved. You can post the log file here or send it directly to me mark @ safe.com

G

Was this issue solved? I am having the same problem. I will try out the workaround suggested by @bruceharold in the meantime. But I just wanted to know if there was a different solution.

+7

Hi @gis_mapper, unfortunately, this issue remains. I've attached a Workspace that handles the truncate-append flow discussed in Bruce's article.

We have not been able to reproduce this reliably in order to fix it. Logging in more recent versions of FME Server has been improved, so if you have an FME Server job log file, preferably with Log Debug mode on, that you could share that would be greatly appreciated! If you aren't comfortable sharing it with the Community, please submit a case to share it with us.

+7

Also, for anyone experiencing this issue randomly, you can avoid duplicates by supplying your own OBJECTID/GLOBALID when inserting features, instead of letting AGOL assign them. That way, a retried write won’t create duplicate rows. You will receive an error instead, but at least it will prevent duplicates.

FME 2023 will introduce an asynchronous writes option for the ArcGIS Online Feature Service Writer, which is unlikely to have the same issues with timeouts and retries.

+16

bruceharold
Contributor
326 replies
1 year ago
13 March 2023

Also, for anyone experiencing this issue randomly, you can avoid duplicates by supplying your own OBJECTID/GLOBALID when inserting features, instead of letting AGOL assign them. That way, a retried write won’t create duplicate rows. You will receive an error instead, but at least it will prevent duplicates.

FME 2023 will introduce an asynchronous writes option for the ArcGIS Online Feature Service Writer, which is unlikely to have the same issues with timeouts and retries.

That is a really good tip!

G

Also, for anyone experiencing this issue randomly, you can avoid duplicates by supplying your own OBJECTID/GLOBALID when inserting features, instead of letting AGOL assign them. That way, a retried write won’t create duplicate rows. You will receive an error instead, but at least it will prevent duplicates.

FME 2023 will introduce an asynchronous writes option for the ArcGIS Online Feature Service Writer, which is unlikely to have the same issues with timeouts and retries.

Hi @mattmatsafe , I have already tried this option and I get a message that the translation failed. So sadly, this is not a solution for me. Or maybe I am adding the wrong parameters. I selected yes for both Use Global IDs and the Preserve Global ID options. Is there an additional option I need to add to make this work? I am trying the change detector but until now it has only deleted my AGOL data but it is not inserting. I selected the Upsert but I get a message saying it is not supported. I will continue looking for a solution. Thanks.

+7

Hi @mattmatsafe , I have already tried this option and I get a message that the translation failed. So sadly, this is not a solution for me. Or maybe I am adding the wrong parameters. I selected yes for both Use Global IDs and the Preserve Global ID options. Is there an additional option I need to add to make this work? I am trying the change detector but until now it has only deleted my AGOL data but it is not inserting. I selected the Upsert but I get a message saying it is not supported. I will continue looking for a solution. Thanks.

@gis_mapper Does the translation fail now because of duplicate IDs? Please feel free to submit a case with your failed job log and we can take a closer look. If you can also share a job log that shows the timeout that leads to duplicates, with Log Debug mode on (turned on in Workbench under FME Options > Translation > Log Message Filter), then we can take a closer look at what is going on and hopefully determine the root cause.

G

Hi @mattmatsafe , I have already tried this option and I get a message that the translation failed. So sadly, this is not a solution for me. Or maybe I am adding the wrong parameters. I selected yes for both Use Global IDs and the Preserve Global ID options. Is there an additional option I need to add to make this work? I am trying the change detector but until now it has only deleted my AGOL data but it is not inserting. I selected the Upsert but I get a message saying it is not supported. I will continue looking for a solution. Thanks.

Hello @mattmatsafe I submitted a ticket and I included the complete log using a change detector. Thanks.

+7

Hi @mattmatsafe , I have already tried this option and I get a message that the translation failed. So sadly, this is not a solution for me. Or maybe I am adding the wrong parameters. I selected yes for both Use Global IDs and the Preserve Global ID options. Is there an additional option I need to add to make this work? I am trying the change detector but until now it has only deleted my AGOL data but it is not inserting. I selected the Upsert but I get a message saying it is not supported. I will continue looking for a solution. Thanks.

Hi @gis_mapper , thanks I see you created a case the other day with the desktop team and that Dan has been working with you. I don't see any timeout or retry in log file provided though, so yours may be a different issue. All HTTP responses from AGOL are coming back 200 successful. If you have a log file where you experienced the timeout, please add it to the case - I'll let Dan know. Thanks!

+7

mattmatsafe
Safer
71 replies
10 months ago
20 June 2023
Best Answer

Hi @gis_mapper and others encountering this issue:

FME Form 2023 (formerly Desktop) has introduced an Asynchronous Writes option to the advanced parameters of the ArcGIS Online Feature Server Writer. Setting this option to Yes should help resolve this issue.

Documentation can be found here: https://docs.safe.com/fme/2023.0/html/FME-Form-Documentation/FME-ReadersWriters/arcgisonlinefeatures/arcgisonlinefeatures_writer.htm

Getting duplicate features when writing to ArcGIS Online

40 replies

Reply

Community Stats

Reply

Community Stats

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded