"Python Exception <UnicodeDecodeError>: 'ascii' codec can't decode byte 0xe4 in position 4: ordinal not in range(128)@Python failed to execute function `StatisticsCalculator.recordFeature' "
I tried to change the Charachter Encoding to utf-8 but then the rows got blank instead.
What is it that I´m doing wrong? Is it the input to StatisticsCalculator that is corrupt? Or is it something from the csv-file?
the StatisticsCalculator transformer is indeed written in Python. Have you looked at the actual data on line 57500 of your CSV file?
I tried to reproduce the error using a small CSV file with accented characters saved as UTF-8, then reading it with FME 2015.0 and the CSV reader set to UTF-8. The StatisticsCalculator did not produce any errors in this case.
A good idea might be to copy the offending lines into a separate and much smaller CSV file for testing purposes. Make sure that you save the file with the same character encoding as the one you specify in the reader in FME.
David
I also was not able to reproduce the error using Japanese characters as input attribute names/values, but I don't think you are doing something wrong.
I think the StatisticsCalculator could have an issue on international character handling, since the transformer is using Python internally as David mentioned, and it's easy to make a bug about encoding in a Python script.
As an interim workaround, try one of these (or both) before calculating.
- rename the attribute to an alias consisting of only ascii characters.
- replace non-numeric attribute values with <null> or empty string, if the attribute may contain non-numeric value.
The actual error reason may be concerning to some factors including the system default encoding, data encoding, reader settings, so it's difficult to specify it.
I would recommend you to contact Safe support, sending your workspace and minimal csv data that reproduces the error.
Takashi
David - I have checked the excel file from which the csv is created and couldn´t find anything wrong there. Maybe there was a problem in the conversion. I tried to read features long after that row but with the same error.
Takashi - I´ll try your workarounds and see if I get any smarter :)
Thank you both.
Daniel
I recommend opening the CSV file using Notepad++ and checking the encoding:
In this case, my CSV file was saved as UTF-8, so I made sure to specify "Unicode 8-bit (utf-8)" on the reader in FME as well. If Notepad++ says ANSI, you should probably set your reader encoding to "Windows Latin-1 ANSI (windows-1252)"
David
Thank you very much
Daniel