I've run into this issue multiple times now, and it's quite serious. My SQL table writer was set to automatic and selected TINYINT
as the datatype. However, one of the values I attempted to write was 5000
, which exceeds the valid range for TINYINT
. As a result, I expected 57 rows to be written, but only 51 actually made it into the database.
Because the system only throws a warning (and buries it in a long list of logs), this kind of data loss can easily go unnoticed unless you're scanning through potentially hundreds of warnings.
The warning I wish to flag or somehow cause the workbench to fail if it occurs:
2025-07-17 14:15:40| 0.5| 0.0|WARN |Microsoft SQL Server Non-Spatial Writer: Failed to parse Unsigned TinyInt' from attribute value
5000' for column `How many kilolitres (kL) of liquid waste was generated?'
Silent data loss is extremely dangerous in data pipelines. How can I flag this better. I am quite surprised FME doesn’t do this by default. I am sure this has impacted many people other than myself, other people on my team have also had this same issue.
Has anyone found a workaround or a way to force an error on this kind of type mismatch?