Can you share how you've set up the DateTimeConverter and what your input dates look like?
Hi,
The Transformer is setup as so. Slightly sensitive information so trying to be aware in the screenshots. As you can see, Pass Through is set to "Yes" but there's still plenty coming through the "Rejected" port
Hi,
The Transformer is setup as so. Slightly sensitive information so trying to be aware in the screenshots. As you can see, Pass Through is set to "Yes" but there's still plenty coming through the "Rejected" port
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Yeah that was the idea in trying to make them standardised (to contain a date or nothing/null) beforehand. FME seems to not pick up on a lot of them though, despite me selecting all the options in the Null Attribute Manager beforehand. Everything looks good, then it gets to the DateTimeConverter and it seems to deviate from what I expect by not finding the Nulls/Missing/Blank attributes
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
I don't think this is what you are seeing, what you are seeing is a Date that doesn't match your input format of %d-%b-%Y which is getting rejected, in the process of rejecting FME sets the date to null, it wasn't null going into the transformer. Do you have some warnings in the log file? Something like
DateTimeConverter: '%d-%b' was successfully parsed, then failed to parse '-%Y'
DateTimeConverter: Failed to evaluate expression '@DateTimeParse(01-May 1980,%d-%b-%Y,repair)'. Result is set to null
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Thanks for your input 
Looking at the Translation Log, there are indeed similar warnings which, it may be the heatwave melting my brain here, but would indicate that even though they're being parsed, still aren't being dealt with optimally
Warning is as follows (and the dates are formatted as, for example, "01-Dec-2015")-
2020-08-12 09:13:26| 25.2| 0.2|INFORM|DateTimeConverter_1: Splitting bulk mode features into features
2020-08-12 09:13:26| 25.2| 0.0|STATS |Storing feature(s) to FME feature store file `E:\temp\wb-cache-Block_Transform_V4_No_D-bNKSak\Main_DateTimeConverter -1 61 fo 1 _lt_REJECTED_gt_ 0 5b5d58d1be6ae7a7ac74bdab8bc98290309ba505.ffsupdating'
2020-08-12 09:13:27| 25.7| 0.5|WARN |DateTimeConverter: '' was successfully parsed, then failed to parse '%d-%b-%Y'
2020-08-12 09:13:27| 25.7| 0.0|WARN |DateTimeConverter: Failed to evaluate expression '@DateTimeParse(NULL,%d-%b-%Y)'. Result is set to null
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
That looks to me like you have values that are the string NULL - not actual null values
Try adding to your Null attribute mapper to handle "Or if Attribute value is:" NULL
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Ok - would removing the NullAttributeMapper help? The reasoning behind that was to make sure any non-date entries were treated as Null and not a mix of missing or otherwise not-a-date format.
Going into the DateTimeConverter, the dates look as follows - some will be null or missing as not all have a value (showing both, the dates and null in this screenshot)
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Removing the NullAttributeMapper, this is the output, which is expected (as I took one that threw an error - there are 2 entries for it, one with a date, one without) and so it still processed it, but still went through the rejected port despite the option to process Nulls/Missing entries
Honing in on the issue, but just need to make it more reliable as it feels hit and miss
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Ooh, no rejected features!
That appears to have fixed it , lesson learnt - "when is a Null not a <null>?"
Thank you, ebygomm! Upvoted and marked as the best response
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
Process Nulls/Missing entries only works on values that are <null> not values that are NULL
If you have values that are NULL you need to use the Null Attribute Mapper to convert them to <null>
This is not a NULL value
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
It's even more fun if you get data like this, one of these is a null value the other is a string '<null>' :-)
If things are passing through the rejected port, it is because they contain values that don't fit the input format. When this happens, the attributes that come out the rejected port have their values replaced by null, even if there was a mix of invalid and valid date formats (not very helpful, i think there is an idea about changing this somewhere).
I'm starting to appreciate up until now, just how nice my source data was from certain teams! :P
