Skip to main content
Gathering Interest

Sorter options for handling null, missing, empty string, non-numeric

Related products:Transformers
rmurphy
+95
  • rmurphy
    rmurphy
  • deborahgayle
  • lewiszhou
  • dbro
  • henry.chu
  • awbriggs
  • jmattison
  • jgollehon
  • tommy_albo
  • colinc
  • sccgis
  • robmackenzie
  • svetrone
  • yaros
  • crutancrutan
  • majpitt
  • gszd55
  • luulhoist
  • rwolframe
  • christianfehrin
  • kateaew
    kateaew
  • mnikosz
  • EleneAtSafe
    EleneAtSafe
  • mkludas
    mkludas
  • hvukasinovic
    hvukasinovic
  • cshepard
    cshepard
  • benzc
    benzc
  • siennaatsafe
    siennaatsafe
  • danilo_fme
    danilo_fme
  • tcrossman
    tcrossman
  • vxn43
    vxn43
  • larue
    larue
  • mattwilkie
    mattwilkie
  • geosander
    geosander
  • kennyo
    kennyo
  • milo89
    milo89
  • tim_wood
    tim_wood
  • pmcpherson
    pmcpherson
  • hlouie
    hlouie
  • gbj1717
    gbj1717
  • fme-academy
    fme-academy
  • kristen
    kristen
  • victor_xvii
    victor_xvii
  • nn_sonomacounty
    nn_sonomacounty
  • jstam
    jstam
  • ccomeau
    ccomeau
  • annab
    annab
  • pkakalika
    pkakalika
  • r5
    r5
  • erooen
  • littlefictions
    littlefictions
  • cwarren
    cwarren
  • garyhk
    garyhk
  • rebeccagiannuzz
    rebeccagiannuzz
  • cdeblonde
  • mvanwie
    mvanwie
  • dino
  • marcandre
  • yongjie
    yongjie
  • tanushree03
    tanushree03
  • chas
  • khirai
    khirai
  • manca_ent
  • marnickcle
    marnickcle
  • timotheebecker
    timotheebecker
  • kalbert
    kalbert
  • dannymatranga
    dannymatranga
  • dos_gis
    dos_gis
  • anari
    anari
  • geoal
    geoal
  • hansh
    hansh
  • olpotosi
    olpotosi
  • angwan
    angwan
  • wicki
    wicki
  • aden
    aden
  • gistogo
  • cpsaltis
  • mapper42
  • iyau
  • garetha83
  • jenniferadhall
    jenniferadhall
  • kierans
  • greg_c
  • mikecw
  • rkortko
  • hogenhoglind
  • rafik.said
    rafik.said
  • karin_h
    karin_h
  • mwawryk
  • rustyrhino
    rustyrhino
  • danderson
  • monika
  • chris_moore
  • chris_s
  • rzimburean
    rzimburean
  • elongvt
    elongvt
  • jack.fifoot
    jack.fifoot
  • gisbalamurugan
    gisbalamurugan
  • shantanu230394
  • duncanrudd

tim_wood
Contributor

Picking up on questions such as
https://knowledge.safe.com/questions/4786/null-sorter.html

and comparing the FME Sorter to Excel, which sorts empty cells to the bottom (for a numeric sort)...

I'd like to suggest adding options to the Sorter or creating a "PedanticSorter" transformer with options allowing the user to sort certain types of values to the end, including nulls, empty strings, non-numeric etc. There could be a drop down box with "sort to end" and "sort to beginning" for each value type. For non-numeric this would apply when you choose to sort numerically but some of the values are not numeric.

A further option could be a "rejected" output port. Then the drop down list for each value type would include "Output via rejected port" as a third option.

9 replies

tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • March 21, 2017
"Sort to top" and "sort to bottom" would be better wording for the choices + "output via rejected port".

 

The default option for each value type would be whatever Sorter does now (usually sort to top), thus maintaining backwards compatibility if the transformer was upgraded old Workspaces.

 


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • March 23, 2017

So the next question is how do we sort these attribute types among themselves if two or more if them exist in the data? For example the attribute includes null AND empty string values.

I would say null comes before empty string if sorting to the top but what about when sorted to the bottom?

Is a hard coded rule acceptable or does it need to be user configurable?


mark2atsafe
Safer
Forum|alt.badge.img+44
  • Safer
  • August 24, 2017

Hopefully this hub transformer does what you need:

https://hub.safe.com/transformers/controllednullsorter


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • May 1, 2018
Thanks @Mark2AtSafe. I haven't had chance to try this yet but as soon as I do I'll feed back :-)

Forum|alt.badge.img
  • August 5, 2021

Another option I have used as a work around uses a combination of NullAttributerMapper-->Sorter-->NullAttributerMapper.

 

Begin with the first NullValueMapper by mapping your NULL, Missing or Empty value to a value you know will be at the top or bottom of your sort (i.e. ZZZZ for a an alphabetic ascending sort). This will replace the value to be used for the Sorter. After your sort load a new NullAttributeMapper and set all of your ZZZZ back to NULL, Missing or Empty. The sorting will remain unchanged after this final mapping.

 

A bit clunky but it works!


mark2atsafe
Safer
Forum|alt.badge.img+44

Now called the AdvancedSorter I believe.


gmbutler2
Contributor
Forum|alt.badge.img+10
  • Contributor
  • December 28, 2023

I actually like the way you think! What I did was simple. I used a tester to look for null, missing, or empty based on specific value that was causing problems. I then create an attribute called sort priority, if it is null I set priority to 1, if it is not null I set priority to 0. Then I apply that to the sorter before sorting that specific value. Thanks for pointing me in a solid direction!


LizAtSafe
Safer
Forum|alt.badge.img+15
  • Safer
  • February 11, 2025
OpenGathering Interest

LizAtSafe
Safer
Forum|alt.badge.img+15
  • Safer
  • February 21, 2025
OpenGathering Interest

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings