FLOW: Latin 1 (ISO/IEC 8859-1) coding on a UTF-8 server machine
Hello.
So we have a FME Flow set up on a Microsoft Azure machine that is causing some problems for our Spanish and German colleagues. More specifically that when their data (mainly KML’s) that contains one of their local language characters is being sent through our Flow tools then the process fails due to “invalid byte ‘local character’ at position 1 of a 6-byte sequence’.
Due to the fact that our users are using diverse data formats then most of our Flow apps use generic readers where they select the format of their input data in a published parameter dropdown that is then feed into the “Input format” of the reader. We are not allowing the feature reader to figure it out itself due to limitations of GeoJson’s and GML’s.
So far I cant see that you can change the character encoding parameters in the Generic reader like you can in specific readers, such as the shapefile one (correct me if I am wrong). So I have instead tried to change the Regional settings of the server machine to Latin (both with enabled “Beta: Use Unicode UTF-8 for worldwide language support” and disabled aswell), but to no prevail.
So questions are two:
Am I somehow able to change the encoding in the workspaces that have the generic reader in them, if so then how?
And if not then what am I doing wrong in changing the encoding of the server machine?
With regards, Hlynur GISus
Page 1 / 1
Firstly, the generic reader in FME Form (Workspaces) supports changing encoding for certain formats, but the option to specify encoding depends on the underlying format you select within the generic reader. For example, text-based formats like CSV or TXT often have an "Encoding" parameter in their reader settings., any Generic format removes the option to specify encoding on the reader.
Secondly, you may wish to edit the workspace, place a tester, from the passed port add Logger with log and record set. Next time you run in FME Flow, you can get a better idea of the character. Sometimes you can use a StringReplacer, replacing each special character (like á, é, í, ó, ú, ü, ñ, etc. with nothing?) or using a Regular Expression Spanish sáéíóúüñÁÉÍÓÚÜÑ] or German äöüßÄÖÜ]
Python and StringPairReplacer also options
here is a similar article in the Community
This is from AI Assist:
The "invalid byte ‘local character’" error in FME Flow usually indicates a character encoding issue, often when reading or writing data that contains non-ASCII or special local characters. This can happen if the data source uses a different encoding (like UTF-8, ISO-8859-1, etc.) than what FME expects.
Here are some steps to troubleshoot and resolve this:
Identify the data format and source. Check if your source (e.g., CSV, database, shapefile) specifies an encoding, and ensure it matches what FME is set to use.
For text-based formats (like CSV), set the correct encoding in the Reader parameters (look for a parameter like "Character Encoding").
If you need to convert encodings, use the TextEncoder transformer to encode or decode text attributes.
Hmm thanks for this.
In regards of the first part then I’m not sure where those encoding parameters you mention should be in a generic reader?
Then for the second then it does not help to add a transformers such as StringReplacer or AttributEncoders to a workspace that fails by kicking off the Reader to begin with..
Does anyone else have some tips?
Personally, I’ve stopped using the Generic reader a long time ago, mostly because of how clunky it is to configure format-specific attributes.
I prefer the following pattern:
Creator → TestFilter (to check the format) → Different FeatureReaders depending on format
That way I can easily configure the different FeatureReaders with format-specific settings, such as the encoding.
Yeah only formats like CSV, database, shape file… have parameter options for encoding.I suspect the formats you are converting don’t have options to change the default encoding. You can read up about https://docs.safe.com/fme/html/FME-Flow/AdminGuide/Configuring_Character_Encoding.htm This required going into the flow machine and Open the web application's deployment descriptor web.xml file, make changes as per the help
So what does the Flow log transaction error with? Including these as attachments helps to narrow down the issue.
Thanks for the help @j.botterill but I’ve decided to dig into the method that @david_r proposed, so far it looks very promising so thank you for that @david_r :)
