This is opened question to FME Community. FME workspaces are very individual. But I believe there is a way quite universal way to test it (FME Testing Framework, rTest, DatasetValidator, Test Custom Transformer). This topic is about data quality. It is crucial!. How do you test your data in FME?
Userlevel 5
+25
- Influencer
- 3345 replies
-
14 November 2022
As you say, it is very individual. In broad terms, I would first check data structure (i.e. the schema), then content.
The one thing that I think is hard to automate is the "sanity check" on the data. Taking a look at the volume and considering whether or not that is an expected amount. If you often process data for local governments you kinda get an idea of the relationship between for example the number of inhabitants vs the number of addresses. I.e. if I process data for a municipality with 100.000 inhabitants but I only have 5.000 addresses I can quite confidently say my data is incomplete.
As you say, it is very individual. In broad terms, I would first check data structure (i.e. the schema), then content.
The one thing that I think is hard to automate is the "sanity check" on the data. Taking a look at the volume and considering whether or not that is an expected amount. If you often process data for local governments you kinda get an idea of the relationship between for example the number of inhabitants vs the number of addresses. I.e. if I process data for a municipality with 100.000 inhabitants but I only have 5.000 addresses I can quite confidently say my data is incomplete.
Okay,
- How do you testing schema and content? Manually?
- How do you prepare expected data based on requirements?
- How do you reporting it? I mean, do you have test cases with execution results passed/failed?
Userlevel 5
+25
- Influencer
- 3345 replies
-
17 November 2022
Okay,
- How do you testing schema and content? Manually?
- How do you prepare expected data based on requirements?
- How do you reporting it? I mean, do you have test cases with execution results passed/failed?
Re. #1: We've used the ChangeDetector on Schema features with some good results. Comparing a supplied schema with an expected schema and then reporting the differences.
Okay,
- How do you testing schema and content? Manually?
- How do you prepare expected data based on requirements?
- How do you reporting it? I mean, do you have test cases with execution results passed/failed?
Okay, I had the same idea (below image). Now my FME Testing Framework is just FME Workbench which you can add (instead of ChangeDetector) with WorkspaceRunner parameters: actual in CSV, expected in CSV and in output you have Test Cases (based on CSV columns) Report in html (pytest) and xls. I plan to do it also for SHP.
Thanks
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.