Skip to main content
Question

SQL non-spatial database update very slow

  • October 17, 2017
  • 3 replies
  • 60 views

Forum|alt.badge.img

The following workflow of comparing tables (in two separate databases) and updating an existing database (SQL non-spatial) is taking for ever. It has been running for 12 hours now to update the database. In 12 hours only 5424 records are compared. I am expecting about 20,000 records to be updated weekly based on update detector comparison. The change detector transformer is in the workbench since I tried that approach as well but failed. How can I make the workflow run faster?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

erik_jan
Contributor
Forum|alt.badge.img+26
  • Contributor
  • October 17, 2017

Is the target table indexed on the key value that you are using in the fme_where?

That could speed up the writing.

Second, could you group the source records some way?

Comparing 1000 with 1000 twenty times is a lot faster than comparing 20000 with 20000 once.

Hope this helps pointing in the right direction.


david_r
Celebrity
  • October 17, 2017

Is the target table indexed on the key value that you are using in the fme_where?

That could speed up the writing.

Second, could you group the source records some way?

Comparing 1000 with 1000 twenty times is a lot faster than comparing 20000 with 20000 once.

Hope this helps pointing in the right direction.

+1 for indexing the table key attribute, super important and can make a huge difference.

ravenkopelman
Safer
Forum|alt.badge.img+1
Agree that an appropriate index on the target table is most likely to be helpful, but it would be interesting to know how fast the translation runs if you disable the writer. In other words, how fast is the reading and change detection on its own?