To simplify this issue, my short term goal is to build a simple module which checks each attribute from a transformer or reader and returns the percentage of nulls for each attribute (vs the total number of records).
for instance something like:
Input:
Attribute1
Attribute2
Attribute3
Attribute4
...
...
Output:
Attribute1
Attribute1-null-count
Attribute2
Attribute2-null-count
Attribute3
Attribute3-null-count
...
...
or something like that...
Again, to clarify, we DO NOT KNOW the Attribute names nor how many attributes there will be.
Ideally, I would build a Custom Transformer, which accepts an unknown number of attributes and performs this calculation / process.
However, Custom Transformers require that the inputs attributes are static.
From all the research Ive done I cannot figure out any way to make this happen.
I thought about a transformer that takes all attributes and combines them into one then they could be broken apart after but nothing seems to exist.
I am open to any suggestions even as far as using FMEServer, Published parameters, whatever it takes, except ideally avoiding Python since this module would be reusable by others therefore avoiding versioning issues..
This is the first step in many Data Quality Checking Modules I would like to generate so my main goal is to find a way to dynamically accept a unknown set of attributes, perform calculations, and return them in their original state with some extra calculated values.
Regards,
Dave
