Skip to main content
Solved

Feature writer - feature types order

  • February 6, 2025
  • 6 replies
  • 52 views

albinepro
Supporter
Forum|alt.badge.img+7

How to securely achieve the desired order of in which are feature types (tables) written in the database? I have some FK dependecies in the target tables and I have somehow discovered that its written from the bottom to the top of the feature types list. If its not written from the parent to the childs it crashes on FK violations.
 


I know there is an idea registered. But so far what are the options? It consider the inserts/updates and also deletes where i would need the inverse order.
Thank you for the suggestions.

Best answer by s.jager

In my experience the only way to be 100% sure that FK-entries are written to the database in the correct order, is to use multiple FeatureWriters after eachother. The first one writes the child-row, then using either an Output-port or the Summary-port you can check if something was actually written, and if it was written you can insert the parent-row. That way you can also keep the essential ID-attribute for the second insert, and you can terminate if the first row was not inserted for some reason, making error-handling a bit neater. Plus, depending on your database, you may need to commit in between first. This way you can set the first FeatureWriter to deal with only 1 Feature per Transaction, which generally causes a commit to take place before the second FeatureWriter.

I found this article (Working with Foreign Keys: Writing Database Tables) extremely helpful, since it deals with this exact situation.

View original
Did this help you find an answer to your question?
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

bwn
Evangelist
Forum|alt.badge.img+26
  • Evangelist
  • February 7, 2025

This is more from observation than from technical documentation, but FME uses workspace optimisation that tries to purge Features held in memory as early as possible by finalising their output to a Writer.

This optimisation will lead to some Feature Types being written in a different order than what is ordered within the Writers or within a FeatureWriter and is dependent on which workflow pathways FME feels it can resolve and finish with “first”.  Your example of going from bottom-to-top is coincidental, in my experience, it can happen in any order due to the above memory usage optimisation techniques used by default.

Some strategies are explained here and similar to what I have generally done, by using things like separate Writers for particular Feature Types, or Transformers that block things from writing until say all features that are needed are present (say into the Input Port of a FeatureHolder) or similarly using a FeatureJoiner that in its behaviour won’t allow anything on the LHS to be output to a downstream writer until all RHS features have been received, or Sorters to block writing and prioritise the order of write within a Writer.

Write Feature Types in a Specific Order


david_r
Evangelist
  • February 7, 2025

If you have a common identifier that links the parent and child features, you can use it as a Group By in the FeatureWriter. This will ensure that the parent and child records are written in a single transaction.

However, be aware that this will require more available memory, as FME will have to cache all the records in memory before writing them. You can mitigate this using the “Complete Groups = When Group Changes”, but it will require that your features arrive in the correct order to avoid problems.

Check the FeatureWriter documentation for more details.


s.jager
Influencer
Forum|alt.badge.img+16
  • Influencer
  • Best Answer
  • February 7, 2025

In my experience the only way to be 100% sure that FK-entries are written to the database in the correct order, is to use multiple FeatureWriters after eachother. The first one writes the child-row, then using either an Output-port or the Summary-port you can check if something was actually written, and if it was written you can insert the parent-row. That way you can also keep the essential ID-attribute for the second insert, and you can terminate if the first row was not inserted for some reason, making error-handling a bit neater. Plus, depending on your database, you may need to commit in between first. This way you can set the first FeatureWriter to deal with only 1 Feature per Transaction, which generally causes a commit to take place before the second FeatureWriter.

I found this article (Working with Foreign Keys: Writing Database Tables) extremely helpful, since it deals with this exact situation.


albinepro
Supporter
Forum|alt.badge.img+7
  • Author
  • Supporter
  • February 10, 2025
s.jager wrote:

In my experience the only way to be 100% sure that FK-entries are written to the database in the correct order, is to use multiple FeatureWriters after eachother. The first one writes the child-row, then using either an Output-port or the Summary-port you can check if something was actually written, and if it was written you can insert the parent-row. That way you can also keep the essential ID-attribute for the second insert, and you can terminate if the first row was not inserted for some reason, making error-handling a bit neater. Plus, depending on your database, you may need to commit in between first. This way you can set the first FeatureWriter to deal with only 1 Feature per Transaction, which generally causes a commit to take place before the second FeatureWriter.

I found this article (Working with Foreign Keys: Writing Database Tables) extremely helpful, since it deals with this exact situation.

I would give a try to disabling and enbling FK constraints for the Feature writers as is in Example 3
But rather I’ll use “SQL to Run before write” and “After”. I dont want to overcomplicate with addition succesive writers as im already dealing with 3 different RDMS in one workspace.


s.jager
Influencer
Forum|alt.badge.img+16
  • Influencer
  • February 10, 2025

disabling and enbling FK constraints

I can’t do that, because I don’t have the rights to do that, so I have to make sure the order is correct. That is usually the fastest way, especially with lots of features, but I only have the rights for DML, DDL is out of the question.

 

 


david_r
Evangelist
  • February 10, 2025
albinepro wrote:

I would give a try to disabling and enbling FK constraints for the Feature writers as is in Example 3
But rather I’ll use “SQL to Run before write” and “After”. I dont want to overcomplicate with addition succesive writers as im already dealing with 3 different RDMS in one workspace.

Disabling FK constraints is somewhat risky unless you’re the sole source of modification to these tables. If FME fails for some reason, there’s a risk of having tables without foreign key constraints until the problem is noticed and fixed. This can even impact query plans (depending on the DB) and lead to a performance degradation.

My personal preference is to import important data to a staging (“scratch”) schema without constraints, where validation occurs. The records are then transferred to the production database using regular SQL and “proper” transaction control for each logical object collection. This is usually very fast as you avoid reading the data back and forth to FME, it all stays within the database. FME is great for orchestrating all of this.

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings