Problem with string lenght

Which version of FME is this? I get the same length for both @StringLength() and the StringLengthCalculator in FME 2020.

It could be linked to the usage of Unicode, where the number of bytes doesn't necessarily correspond to the number of characters. Notably, the letter "Ä" in Unicode is represented by the two bytes c3 + 84 (hex). If an algorithm doesn't account for this, or if it doesn't know that it's a multibyte string, it might return the byte length (=2) rather than the character length (=1) for the string "Ä".

If you send the string to e.g. the Logger in FME you should be able to see the encoding associated with the attribute, e.g.

This is spot on with the developer comments on the tracked issue. The long winded explanation can be found here: http://unicode.org/faq/char_combmark.html#7

It seems the expectation would be to count graphemes, (what is rendered on the screen) as a single logical character rather than bytes/code units.

HI @kasparlov,

Sorry to hear you are running into this issue.

This has been reported in the past and is currently being tracked in our system as FMEENGINE-48508 (also posted on this idea). For the time being, please continue using the StringLengthCalculator transformer.

If you'd like to be added as a contact for the tracked issue, please submit a case and reference FMEENGINE-48508 or let me know and I can create a case on your behalf. Additionally, please be sure to upvote and comment on the linked idea!

Which version of FME is this? I get the same length for both @StringLength() and the StringLengthCalculator in FME 2020.

It could be linked to the usage of Unicode, where the number of bytes doesn't necessarily correspond to the number of characters. Notably, the letter "Ä" in Unicode is represented by the two bytes c3 + 84 (hex). If an algorithm doesn't account for this, or if it doesn't know that it's a multibyte string, it might return the byte length (=2) rather than the character length (=1) for the string "Ä".

If you send the string to e.g. the Logger in FME you should be able to see the encoding associated with the attribute, e.g.

Hi @chrisatsafe,

The answer may be found in the source dataset, which format is it? Are there any encoding options on the reader? If the source is the HTTPCaller, make sure to specify the result encoding rather than letting FME guess. Encoding guesses may take the OS settings into account (all things depending) and that could potentially explain the issues.

Encoding is set to UTF-8 in the reader

Then try the tip about sending the features to the Logger directly before the StringLength calculation, to check that it still says UTF-8 on those attributes. If not, then some transformer did something with the attribute encoding along the way.

I found the problem. In the Region Settings on Windows 10 there is a checkbox "Beta: Use Unicode UTF-8 for worldwide language support" which was checked. When I unchecked the option the string length was calculated correctly.