Ok , I found the offending character for you..
if you check the field values in the Visual Preview window or closely in the Data Inspector you'll notice this character (?) at the end of the several strings.
So, to remove this invalid character(s), use the String Replacer, and set the mode to Replace Regular Expression. see snapshot below.
Thank you, @samisnunu so much. That was amazing.
I had of course seen the ? character. I know that this is a unicode catch-all U+FFFD, used to replace an unknown, unrecognized or unrepresentable character. And I did try to replace that
as text (which it is not) and that failed. Hence my appeal for help. Mode
Regular Expression did the trick, at least this time.
I also believed that the ? was generated in FME when reading the shapefile, but it probably originated in the supplying system.
Thanks once again. This was a useful lesson.
/Mats,E
Hello everyone, I would like to reopen this post.
I find a similar problem with a csv file.
This file has columns with an encoding problem with characters like â; é; ô(which are respectively â, é, ô)
When I try to load thi file on my Postgis DB FME gives the error:
'ERROR: invalid byte sequence for encoding "UTF8": 0xc3 0x20
If I'm not mistaken, 0xc3 0x20 translates to (Ã + white space) but I couldn't find these two characters on Data Inspector...
For the moment I tried to transform the file with StringReplacer for the characters and AttributeEncoder for translate from ISO-8859-1 to UTF-8.
Data Inspector shows that all columns are in UTF-8 and no weird characters, but the same error remains.
My BD has UTF-8 encoding, Collation and Character type in English_United States.1252.
Have you any idea how can I handle this error?
Thanks!
Hello everyone, I would like to reopen this post.
I find a similar problem with a csv file.
This file has columns with an encoding problem with characters like â; é; ô(which are respectively â, é, ô)
When I try to load thi file on my Postgis DB FME gives the error:
'ERROR: invalid byte sequence for encoding "UTF8": 0xc3 0x20
If I'm not mistaken, 0xc3 0x20 translates to (Ã + white space) but I couldn't find these two characters on Data Inspector...
For the moment I tried to transform the file with StringReplacer for the characters and AttributeEncoder for translate from ISO-8859-1 to UTF-8.
Data Inspector shows that all columns are in UTF-8 and no weird characters, but the same error remains.
My BD has UTF-8 encoding, Collation and Character type in English_United States.1252.
Have you any idea how can I handle this error?
Thanks!
@esteban.carcamo I would expect the AttributeEncoder to work in this case. As a workaround, you can define the encoding of the data in the CSV Reader itself when you add it to the workspace. When adding the CSV Reader, click the parameters button and then change the Character Encoding to Unicode 8-bit (utf-8).
Testing this myself, I was able to write the â, é, ô characters to a PostgreSQL table. Let me know if this works for you. (Note: I am using FME Workbench 2022.2)
