Question

Removing duplicate features AND geometry

  • 2 March 2022
  • 8 replies
  • 118 views
T
Badge

I am using the DuplicateFilter to remove features with the same attribute. It does work well but I am having an odd issue with the geometries behind each feature.

The features that gets filtered out somehow 'still exists in the background'. It doesn't get counted as a normal feature if loaded in GIS, nor appear as one, but when using transparent symbology it is visible, as if it hasn't been fully removed. Just the attribute seem to be removed. The filtered out feature still seem to exist hidden in the dataset (a shapefile in this case).

 

How can geometries be filtered out properly with their attributes? duplicates

8 replies

nielsgerrits
Rank
Userlevel 6
Badge +32

Without sample data or a printscreen it is hard to say what your issue is, but I often use a Matcher to find duplicate features.

T
Badge

Without sample data or a printscreen it is hard to say what your issue is, but I often use a Matcher to find duplicate features.

sorry, here is a screenshot. The tiles do show as still there via the transparency. But they aren't actually there. These are the one showing filtered out as Duplicate.

T
Badge

The GeometryValidator class those features as Failed.

QGIS classes them invalid as self-intersections.

nielsgerrits
Rank
Userlevel 6
Badge +32

The GeometryValidator class those features as Failed.

QGIS classes them invalid as self-intersections.

Ah, then it got probably more than 4 segments. If you select one of those in the visual preview and look at the coordinates in the Feature Information window, you will probably see why it is invalid.

T
Badge

Ah, then it got probably more than 4 segments. If you select one of those in the visual preview and look at the coordinates in the Feature Information window, you will probably see why it is invalid.

dupicates2But I believed those errors come from the DuplicateFilter. These features are fine before the DuplicateFilter. It looks like FME is removing duplicate features in a dirty way, it's not straightforward anyway.

nielsgerrits
Rank
Userlevel 6
Badge +32

Ah, then it got probably more than 4 segments. If you select one of those in the visual preview and look at the coordinates in the Feature Information window, you will probably see why it is invalid.

Hard to say without data but I would be surprised. The DuplicateFilter does not modify features, it filters them. If you check the output of the transformers upstream of the DuplicateFilter you will find the same features.

M
Badge +2

@ts_84​ It looks like just an issue with the way you're using Visual Preview/Feature Caching. Just select If you select the transformer, data form all output ports is displayed. In the Visual Preview / Table View you'll have a drop down list of the different caches you have selected. On the Display Control icon, you can toggle on the Display Contol panel and select which caches to view

OR

If you just select the green icon on the Unique port, only those cached features will be displayed. In the Visual Preview / Table View you'll

T
Badge

So I think the issue is due to me using the Feature Merger in a bad way before the DuplicateFilter. Those 3 GeometryValidator show where the error happens.

imageMaybe I should use the Matcher instead.

Thanks everyone for their time and answers

Reply


Community Stats

28,707
Posts
109,477
Replies
36,551
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.