Question

Can I treat CSV reader type mismatches as error instead of warning?


I have a scenario where I have a CSV reader which reads in values of specific types. On occasion there is unexpected data in the input that does not match the type (such as a negative number where a UInt32 is expected, or character data where numeric is expected). The CSV reader generates a warning message and substitutes the offending value for missing or null, but I would prefer if it generated an error so I could terminate translation and look at the data myself to see if it is either bad data (which I need to fix) or if my input types are incorrect and need to be expanded.

 

Reviewing the logs every time is possible of course, but sometimes the processes are automated, and sometimes I need the workspace to terminate on error before it gets to the point where it writes the final data. In these cases using the logs to check warnings isn't great.

 

I can set the CSV reader to read in only string data, but I still want the type checking as a first data validation step. Ideally I would be able to have FME treat these sorts of type mismatches as errors instead of warnings but I don't see this option in Workbench. Is it possible somehow, or does anyone have other possible solutions I could try for validation of CSV input data?


2 replies

Userlevel 2
Badge +11

Hi @ubernatural Try an AttributeValidator (and/or a Tester) along with a Terminator.

Hi @ubernatural Try an AttributeValidator (and/or a Tester) along with a Terminator.

Hi @danatsafe​, I thought about validating the attributes but by the time the features hit the validator they've already run through the CSV reader. In my case, the source can contain legitimate null/empty/missing attributes. I can't differentiate between null values that are valid and those that are the result of a type conversion failure. If there were something similar to the changes list for the ChangeDetector (for example) but for type conversion errors I'd be able to look for that in the Tester or AttributeValidator, but I don't see anything I can use there.

Reply