Skip to main content
Solved

Writing to an SDE dramatically slower than writing to a file geodatabase?

  • September 3, 2019
  • 2 replies
  • 285 views
michael_m
Contributor
Forum|alt.badge.img+1

I have a workbench that I regularly use to read from an Oracle Locator database and write to an ESRI file geodatabase. It writes about 55 feature classes to this geodatabase (approximately 8 million features total) and takes about 90 minutes to complete.

I have just had an SDE made available to me so I made a copy of my original file gdb workbench and have been changing it to write to the SDE instead. The problem I'm having is the workbench now writes extremely slowly; it took 90 minutes to write a single feature class with 168,000 features in it (at that rate I estimate it would take two or three days to write all 8 million features to the SDE).

Has anybody experienced anything like this or have any idea what I can do to speed it up? If it makes a difference I believe the SDE is running on some version of SQL Server.

Thanks,

Mike

Best answer by virtualcitymatt

Hi @michael_m,

 

I'm not sure if you still have this problem, however, there is this article which might be useful: https://knowledge.safe.com/articles/165/improving-performance-when-working-with-esri-geoda.html

 

 

In General I would say SDE is a bit slower than File Geodatabase, however, what you are describing is painful. One thing which you can check in the logfile is the difference between CPU usage vs FME Session duration. If the CPU is very low compared to Session Duration then it means FME is waiting. This could be due to slow network or a particular configuration of the SDE itself (e.g., versioning) or the writer configuration (transaction type).

 

 

Another thing which I've seen is due to limited permissions - As I remember there were cases where the Feature type had it's table handling to Truncate Existing, however, the user didn't have permission to truncate so and edit session was started and each feature was deleted one by one....This doesn't sound like you though because from the sounds of it you are just trying to create them and fill them up.

 

 

If you are really not making progress then I would get as much information about your environment as possible form IT (FME version, 64 bit vs 32 bit, ESRI version, underlying Database version etc) and send a reproduction to Safe Software along with the logfiles and workspace you have. They can try and reproduce your issue or, at least, rule FME out of the picture.

 

 

Good luck.
View original
Did this help you find an answer to your question?

2 replies

virtualcitymatt
Celebrity
Forum|alt.badge.img+36
  • virtualcitymattCelebrity
  • Celebrity
  • Best Answer
  • September 25, 2019

Hi @michael_m,

 

I'm not sure if you still have this problem, however, there is this article which might be useful: https://knowledge.safe.com/articles/165/improving-performance-when-working-with-esri-geoda.html

 

 

In General I would say SDE is a bit slower than File Geodatabase, however, what you are describing is painful. One thing which you can check in the logfile is the difference between CPU usage vs FME Session duration. If the CPU is very low compared to Session Duration then it means FME is waiting. This could be due to slow network or a particular configuration of the SDE itself (e.g., versioning) or the writer configuration (transaction type).

 

 

Another thing which I've seen is due to limited permissions - As I remember there were cases where the Feature type had it's table handling to Truncate Existing, however, the user didn't have permission to truncate so and edit session was started and each feature was deleted one by one....This doesn't sound like you though because from the sounds of it you are just trying to create them and fill them up.

 

 

If you are really not making progress then I would get as much information about your environment as possible form IT (FME version, 64 bit vs 32 bit, ESRI version, underlying Database version etc) and send a reproduction to Safe Software along with the logfiles and workspace you have. They can try and reproduce your issue or, at least, rule FME out of the picture.

 

 

Good luck.
michael_m
Contributor
Forum|alt.badge.img+1
  • michael_mContributor
  • Author
  • Contributor
  • September 30, 2019
virtualcitymatt wrote:

Hi @michael_m,

 

I'm not sure if you still have this problem, however, there is this article which might be useful: https://knowledge.safe.com/articles/165/improving-performance-when-working-with-esri-geoda.html

 

 

In General I would say SDE is a bit slower than File Geodatabase, however, what you are describing is painful. One thing which you can check in the logfile is the difference between CPU usage vs FME Session duration. If the CPU is very low compared to Session Duration then it means FME is waiting. This could be due to slow network or a particular configuration of the SDE itself (e.g., versioning) or the writer configuration (transaction type).

 

 

Another thing which I've seen is due to limited permissions - As I remember there were cases where the Feature type had it's table handling to Truncate Existing, however, the user didn't have permission to truncate so and edit session was started and each feature was deleted one by one....This doesn't sound like you though because from the sounds of it you are just trying to create them and fill them up.

 

 

If you are really not making progress then I would get as much information about your environment as possible form IT (FME version, 64 bit vs 32 bit, ESRI version, underlying Database version etc) and send a reproduction to Safe Software along with the logfiles and workspace you have. They can try and reproduce your issue or, at least, rule FME out of the picture.

 

 

Good luck.

Thank you, @virtualcitymatt. It did turn out to be the setting for "Transaction Type". I changed that from "Versioned Edit Session" to "Transaction" and the overall write time fell back to about two hours.

Reply


Helpful Members This Week

Community Stats

31,350
Posts
118,922
Replies
38,966
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.