Skip to main content
Question

FeatureWriter slow writing back to FGDB


I have a workbench that is reading from an already populated FGDB with 140 Feature Classes. I am trying to add an extra column to the majority of those feature classes and then populate the column for each row with a URL and ID to match the row. I am finding that it is taking 6 minutes to write back 60,000 rows and considering there are over 10 million to update that is an unacceptable timescale.

I am creating the new attribute and a non spatial index with a python caller. Feature Reader to get the ID of each row. AttributeCreator and Manager to create the URL and merge it with the ID. Finally an AttributeFilter to the FeatureWriter. In the FeatureWriter I am doing an Update and using PrimaryKey columns for the row selection. I would appreciate any thoughts on how I might quicken the process.

8 replies

  • Author
  • April 3, 2020

I am running 2019.2


david_r
Celebrity
  • April 3, 2020

Would it be possible to try using an DatabaseUpdater for the updates, rather than the FeatureWriter? It might be quicker in some instances.

Also, check and double-check that all the columns used in where-clauses are properly indexed, that could potentially make a huge impact.


erik_jan
Contributor
Forum|alt.badge.img+18
  • Contributor
  • April 3, 2020

Another option would be reading the original FGDB, adding the new attribute and writing a new FGDB, using the INSERT mode.

Then renaming the new FGDB to match the old name (or use the old name for the FGDB in a different location and move after processing).

INSERT mode will always be a lot faster than UPDATE.

Hope this helps.


  • Author
  • April 3, 2020
david_r wrote:

Would it be possible to try using an DatabaseUpdater for the updates, rather than the FeatureWriter? It might be quicker in some instances.

Also, check and double-check that all the columns used in where-clauses are properly indexed, that could potentially make a huge impact.

Thank you for your suggestion. I have not come across the DatabaseUpdater, it does seem to be quicker. I will test it with some of the larger Feature Classes and see how it runs.


  • Author
  • April 3, 2020
erik_jan wrote:

Another option would be reading the original FGDB, adding the new attribute and writing a new FGDB, using the INSERT mode.

Then renaming the new FGDB to match the old name (or use the old name for the FGDB in a different location and move after processing).

INSERT mode will always be a lot faster than UPDATE.

Hope this helps.

Thank you for your comment, There is surprisingly an improvement, I will try this with larger feature classes.


Forum|alt.badge.img+2

@markb I like @erik_jan ' s suggestion of avoiding updates if you can. Once you have to use an Edit Session with a Geodatabase writer things slow down dramatically. If you have simple feature classes (i.e. point,line,polygon) and no relationships or annotations etc, then the File Geodb API might also speed things up for you. And you could try that in update mode as well


  • Author
  • April 7, 2020
markatsafe wrote:

@markb I like @erik_jan ' s suggestion of avoiding updates if you can. Once you have to use an Edit Session with a Geodatabase writer things slow down dramatically. If you have simple feature classes (i.e. point,line,polygon) and no relationships or annotations etc, then the File Geodb API might also speed things up for you. And you could try that in update mode as well

@markatsafe Thank you for your reply, Are you suggesting using the Esri Geodatabase (File Geodb Open API) instead of the Esri Geodatabase (File Geodb)? I always thought it was better to use the latter?

It is certainly a lot quicker writing out to a new FGDB, thanks to you and @erik_jan


Forum|alt.badge.img+2
markb wrote:

@markatsafe Thank you for your reply, Are you suggesting using the Esri Geodatabase (File Geodb Open API) instead of the Esri Geodatabase (File Geodb)? I always thought it was better to use the latter?

It is certainly a lot quicker writing out to a new FGDB, thanks to you and @erik_jan

@markb Each Geodb reader/writer, Esri Geodatabase (File Geodb Open API) vs. Esri Geodatabase (File Geodb) , has it's place.

Esri Geodatabase (File Geodb Open API) by-passes Esri licensing requirements and ArcObjects so is more accessible and can be faster. But is limited to simple features (think SHAPE files in Geodb)

Esri Geodatabase (File Geodb) gives access to all the feature classes an geometry types available in a Geodatabase and supported by FME.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings