I'm not sure what is happening, but when I inspect a CSV the values in the CSV are not equal to the values read in by the CSV reader in FME Desktop 2021.1.0.0 Win64.
Actual CSV Values
FME Inspector
As you can see it is not reading the first few digits of the easting values. Any ideas as to what might be the issue? My first thought was some sort of encoding problem, but wasn't sure. I've attached the CSV file in question.
This has to do with the automatic interpretation of the type. I believe this is a issue in the newer versions, can't reproduce this in 2019.2.1 where all fields are set to string.
It set easting to time. Switch to manual and change it to real64 will fix your problem.
This has to do with the automatic interpretation of the type. I believe this is a issue in the newer versions, can't reproduce this in 2019.2.1 where all fields are set to string.
It set easting to time. Switch to manual and change it to real64 will fix your problem.
Oh crazy, I've never seen this
This has to do with the automatic interpretation of the type. I believe this is a issue in the newer versions, can't reproduce this in 2019.2.1 where all fields are set to string.
It set easting to time. Switch to manual and change it to real64 will fix your problem.
That did the trick! I must have overlooked that. Thanks again @nielsgerrits
Hi @Chris Warren,
We tried to improve our CSV reading by automatically interpreting date types, but as you have found, coordinate values can be mistaken for FME date types. We have changed this in our upcoming FME 2022 betas so that only ISO formatted dates will be automatically interpreted.
I have tried the solution mentioned above (changing the field format to real64) and it didn't make any difference - the values were still displayed incorrectly.
I have tried the solution mentioned above (changing the field format to real64) and it didn't make any difference - the values were still displayed incorrectly.
Can you share a small sample of your data?
Yes (see attachment).
If you process this data, e.g. create a new csv from the existing csv, the output file will display the easting and northing values incorrectly in Visual Preview or Data Inspector (e.g. if you right click and select "View Written Data") even after you have set the input (reader) field formats correctly and have made sure the output field formats match them. You can force FME to display the correct values by going into Data Inspector and adding the dataset via the toolbar buttons and manually setting the field formats to the correct types (e.g. real64) - by default it still tries to interpret them as time format.
Yes (see attachment).
If you process this data, e.g. create a new csv from the existing csv, the output file will display the easting and northing values incorrectly in Visual Preview or Data Inspector (e.g. if you right click and select "View Written Data") even after you have set the input (reader) field formats correctly and have made sure the output field formats match them. You can force FME to display the correct values by going into Data Inspector and adding the dataset via the toolbar buttons and manually setting the field formats to the correct types (e.g. real64) - by default it still tries to interpret them as time format.
I agree, this is not a good thing.
@mark2atsafe can you pass this on to the support department? As you can see here, when letting the CSV reader automatically determine the column types the preview shows wildly incorrect values because it's trying to interpret certain numbers as dates.
The FME Academy module "Turn Coordinates Into Geometry" also mentions this and to be honest, as a trainer, I find this an awkward moment in our courses
I agree, this is not a good thing.
@mark2atsafe can you pass this on to the support department? As you can see here, when letting the CSV reader automatically determine the column types the preview shows wildly incorrect values because it's trying to interpret certain numbers as dates.
The FME Academy module "Turn Coordinates Into Geometry" also mentions this and to be honest, as a trainer, I find this an awkward moment in our courses
What versino are you running. I couldn't see this in FME 2021.0. Is this a 2021.1 thing? Did you try it on FME 2022 beta?
I agree, this is not a good thing.
@mark2atsafe can you pass this on to the support department? As you can see here, when letting the CSV reader automatically determine the column types the preview shows wildly incorrect values because it's trying to interpret certain numbers as dates.
The FME Academy module "Turn Coordinates Into Geometry" also mentions this and to be honest, as a trainer, I find this an awkward moment in our courses
2021.2 (build 21784) on Mac
We are using FME 2021.1.0.0. I haven't tried 2021.0 or 2022 as I do not have these available to me.
We are using FME 2021.1.0.0 on Windows. I do not have access to 2021.0 or 2022 beta.
@donnaviney @Chris Warren This has been fixed in FME 2022. FME will now defer to x/y northing/easting. We've also switch the default Input Date Format to ISO (under Schema Generation) which is less likely to detect floats as date/time.
If you are using an earlier version of FME, and use CSV files quite a bit, then set the Date Input Format as ISO and then save that as the default Preset
Hi.
Please can you explain how to change the default input date format. I am not sure where to find "schema generation".
Thanks.
Donna.
Thanks very much Mark. That seems to be working.
My workaround for this was to load the data in as text-line, then use attribute splitter and manually bash in column names. Not ideal, but it's a workaround.
Suggestions above did not fix the issue in FME 2021.2.
