Skip to main content
  • EN

I have a nine element, pipe-delimited attribute. These elements can be a text string or a blank space. I want to replace duplicate text elements with a blank space and leave blank elements alone. I.e. have it come out with eight delimiters and nine elements when I put it back together.

ListDuplicateRemover gets rid of the unwanted text elements, but also the blank ones.

 

Example:

|||Carnegie Drive|Carnegie Drive||Dunfermline||Ky12 7bd

when run through AttributeSplitter and ListDuplicateRemover becomes:

|Carnegie Drive|Dunfermline|Ky12 7bd

But I want it to be ' ||||Carnegie Drive||Dunfermline||Ky12 7bd' (with a leading space)

I.e., just remove the duplicate instance of 'Carnegie Drive' and replace it with a blank space.

Thoughts? Thanks

Explode the list, use a duplicate filter to set the value of anything coming out of the duplicate port to blank then rebuild the list?

 

Edit: just noticed you need the first element to be replaced with a blank rather than the second which makes it a bit more complicated

Are your duplicates always going to be adjacent to each other?

Are your duplicates always going to be adjacent to each other?

Not necessarily. I think at this point, I would just settle for no duplicates/right number of elements.

Am trying your suggestion - ListExploder, DuplicateFilter, using AttributeManager on the duplicate port to set value to ' ' and plugging it all into ListConcatenator. Currently, though, LC cannot see any list attributes...

remove_duplicates_from_list.fmw

Not necessarily. I think at this point, I would just settle for no duplicates/right number of elements.

Am trying your suggestion - ListExploder, DuplicateFilter, using AttributeManager on the duplicate port to set value to ' ' and plugging it all into ListConcatenator. Currently, though, LC cannot see any list attributes...

After exploding you no longer have a list, you need to rebuild the list before concatenating

After exploding you no longer have a list, you need to rebuild the list before concatenating

Yes, of course. Where's the facepalm emoji on this thing...?

If i could replace 8 transformers with one python caller though I'd probably go down that route

import fme
import fmeobjects

def replace_duplicates_with_blanks(feature):
    text = feature.getAttribute('text')
    list = text.split("|")
    list.reverse()
    newlist= ]
    for i, val in enumerate(list):
        if val in newlist:
            newlist.append("")
        else:
            newlist.append(val)
    newlist.reverse()
    separator = '|'
    newtext = separator.join(newlist)
    feature.setAttribute('newtext',newtext)

Reply


Community Stats

30,159
Posts
114,480
Replies
37,806
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.