Skip to main content
  • EN
A tool in FME to remove duplicates,

 

 

I couldn’t figure out which tool can remove duplicates. For example, in case we have two lines that are identical (one on top of another), then which tools can remove this duplicate?

 

 

For example, in the screenshot below we have 4 lines that are identical, then how to remove three of them and to leave one?

 

 

 

 

Thank you

 

 

Best

 

 

Jamal
The Matcher transformer would do this for you. If you take the output of the SingleMatched and NotMatched ports then you would - in effect - be dropping duplicates.

 

 

Hope this helps

 

 

Regards

 

 

Mark

 

 

Mark Ireland

 

Product Evangelist

 

Safe Software Inc
Many thanks Mark for the help,

 

 

It appears that the “matcher” couldn’t catch the duplicates. It keeps the original number as is despite the fact that there are a plenty of duplicates.

 

 

 

 

 

 

 

What might be the issue here?

 

 

The dwg file:

 

http://www.mediafire.com/download/xhrc6kchbacx0s0/Parcels.dwg
Hi Jamal,

 

Have a look at the "Lenient Geometry Matching" parameter in the Matcher.

 

Takashi
Thanks Takashi,

 

 

I couldn’t figure out precisely how the matcher works and how does it remove the duplicates.

 

 

1. I wanted to remove the duplicates

 

 

 

2. And then to convert lines to polygons.

 

 

 

 

 

This doesn’t appear to work with the “matcher” and “GeometryCoercer”.

 

 

The polygons fail to be written in the polygon feature class?

 

 

What might be the issue here?
The Matcher seems to work properly. The problem is on the GeometryCoercer, it will not transform lines into polygons. Consider using the AreaBuilder instead.
Many thanks Mark and Takashi for the help,

 

 

It appears to work fine now as I have adjusted the workbench based on your valuable comments.

 

 

The Matcher transformer would do this for you. If you take the output of the SingleMatched and NotMatched ports then you would - in effect - be dropping duplicates.

 

 

Hope this helps

 

 

Regards

 

 

Mark

 

 

Mark Ireland

 

Product Evangelist

 

Safe Software Inc
It would be nice if the transformer had a port for "matched except one" so you can remove all but one duplicate record easily. Currently, if I understand the transformer, you have to delete the Matched and then add back in the SingleMatched. Matched=SingleMatched+MISSING port.

 

 

 

Not sure which tool can remove the duplicates shown in the screenshot below

 

Line#3 and line #4 are duplicated in the marked part

 

Clip_90 

 

Clip_89 

 

 

 

 

 

Not sure which tool can remove the duplicates shown in the screenshot below

 

Line#3 and line #4 are duplicated in the marked part

 

Clip_90 

 

Clip_89 

 

 

 

 

If they match exactly I'd try the Intersector or LineOnLineOverlayer. Both have the option to generate a list with the attributes of the original overlapping features so you can work with those.

So basically, in your screenshot, you can remove the Chopper and Matcher and it should work.

 

Not sure which tool can remove the duplicates shown in the screenshot below

 

Line#3 and line #4 are duplicated in the marked part

 

Clip_90 

 

Clip_89 

 

 

 

 

Hello @jamal​ , this article has a few alernative approaches, it may be worth taking a peek! Data QA: Identifying Duplicate Features with FME.

Right. Intersector or LineOnLineOverlay can remove the duplicates as per the screenshot below. The duplicate segment gets in a separate object. All can be joined with linecombiner tool

 

Clip_1140 

Clip_1141 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Reply


Community Stats

30,176
Posts
114,541
Replies
37,835
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.