Improve Sorter to handle accented characters
+22- Safer
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+19- Influencer
Possible workaround:
- Copy sort value into temporary attribute
- Remove all accents using PythonCaller call to 'removeAccents':
import fmeobjects import unicodedata as ud
def rmdiacritics(char): ''' Return the base character of char, by "removing" any diacritics like accents or curls and strokes and the like. ''' desc = ud.name(unicode(char)) cutoff = desc.find(' WITH ') if cutoff != -1: desc = desc[:cutoff] return ud.lookup(desc)
def removeAccents(feature): attribute_list = ("name", "type", "state") # Modify as needed for attrib in feature.getAllAttributeNames(): if attrib in attribute_list: value = feature.getAttribute(attrib) if value: value = unicode(value) new_value = ''.join([rmdiacritics(char) for char in value]) feature.setAttribute(attrib, new_value)
- Sort on accent-less attribute
+10
For now I adjust my date notation to yyyymmdd, so I can sort by number, but it would be nice to add this to the sorter.
This really needs to be implemented, as it is the Sorter is pretty much useless for non-English languages.
If the performance suffers too much, maybe consider having two different alphanumeric sorts, one based on byte order and one using the locale sort order (with full Unicode support).
Maybe also support custom sort orders by language group, see https://msdn.microsoft.com/en-us/library/windows/desktop/dd374060%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
Agreed, as long as local and language-specific conventions are adhered to. For example, in some languages, letters with accents/diacritics are considered separate letters that may or may not follow their base letter. For example:
- In German, letters with umlaut are treated like their non-umlauted versions
- In Polish ?, ?, ?, ?, ?, Ó, ?, ?, and ? are all separate letters in the alphabet that follow their base letter. The same rule applies to the Spanish Ñ.
- In Swedish, there are three extra vowels placed at the end (..., X, Y, Z, Å, Ä, Ö).
https://en.wikipedia.org/wiki/Alphabetical_order#Language-specific_conventions
+22- Enthusiast
Why am I able to sort correctly my accents into Data Inspector but not with the Sorter?
I'm on FME 2018.1.
+2- Contributor
Using French for products so ability to select language would be good for sorter as noted, but also numeric settings. French uses a comma instead of decimal for numeric values ( (i.e. 10,12 vs 10.12 in English).
Login to the community
No account yet? Create an account
An FME Account is required to contribute
LoginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.