After the FME ETL process is completed, is there a way to validate the results automatically?
Question
How to validate results of FME ETL process?
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+51- Celebrity
Hi @fantomfoto,
In my opinion you need to create another Workspace to validate your Writer:
- Schema
- Attributes
- Geometry
Thanks,
Danilo
Hi @fantomfoto
you definitely can do some automatic validation, e.g. you can check if a table has indeed been written into your database or the number of files output into a folder matches the expected number. The validation depends on your data and what might go wrong or what you would like to confirm did not go wrong. If you describe your scenario in more details, I am sure community will be happy to share ideas and suggestions.
+14- Safer
For a general idea of the types of validation you can implement into your workflow, please see this Knowledge Center Article.
And, generally speaking, you could leverage FeatureWriter / FeatureReader in place of the native Reader / Writers in FME to have your ETL and validation occur in the same workspace – but it can just as easily be split into two workspaces. Note that this validation process itself will more or less have to read all the data back into FME, so when dealing with very large datasets it may impart a measurable overhead (especially those datasets where you cannot leverage filters or SQL statements to limit the number of features actually coming into the workspace).
+51- Celebrity
For a general idea of the types of validation you can implement into your workflow, please see this Knowledge Center Article.
And, generally speaking, you could leverage FeatureWriter / FeatureReader in place of the native Reader / Writers in FME to have your ETL and validation occur in the same workspace – but it can just as easily be split into two workspaces. Note that this validation process itself will more or less have to read all the data back into FME, so when dealing with very large datasets it may impart a measurable overhead (especially those datasets where you cannot leverage filters or SQL statements to limit the number of features actually coming into the workspace).
+14- Safer
Hi @RylanAtSafe this article that you posted above is very interesting.