• Home
  • Forums
  • FME Form
  • Data
  • Can FME load a MapInfo .TAB file and perform a task to remove duplicate objects? (Essentially flattening stacked objects)
Question

Can FME load a MapInfo .TAB file and perform a task to remove duplicate objects? (Essentially flattening stacked objects)

  • 25 July 2013
  • 6 replies
  • 0 views
N
Hello,

 

 

I am going to be upgrading to FME 2013 today, and after doing so, I was wondering if anyone could lend a hand in telling me if FME can provide what I'm looking to do.

 

 

1. We stored all data in a MapInfo .TAB file (flat files) for each customer

 

2. Customers will give us new data on a weekly basis; however, the data being given includes everything from the first submission, and then perhaps some additions that are legitmate.  In doing step 2, the customer is actually creating duplicate objects stacked on top of the first upload.

 

 

My question is can FME run a process to find and eliminate all those duplicate objects so that only one remains (since the others are useless).

 

 

Thanks for any insight in advance.

 

Nick

6 replies

takashi
Rank
Userlevel 2
Badge +17
Hi Nick,

 

 

If every object has an identifier as its attribute, the DuplicateRemover can be used. Otherwise, you can use the Matcher to find duplicate geometric shapes.

 

 

Takashi
N
Takashi,

 

 

I will give this a shot and post my results.  

 

 

Thanks for the quick resposne.

 

Nick
fmelizard
Rank
Userlevel 4
Badge +13
Hi,

 

 

In case no such unique id exists you can try the following: CRC

 

 

Hope this helps,

 

Itay
N
Hi Guys,

 

 

Thanks for the advice. However, since I am so beginnier to using FME, I was curious how I begin.  I started updating to FME 2013 SP2, opened FME Workbench, then chose "Generate Workspace".

 

 

I was able to get to the following setup (seen below image):

 

 

 

My question now becomes; how do I start to introduce the "Matcher"?  Is that a Transformer, or Inspector, etc. ?  I do understand that the "DuplicateRemover" is not what I want to use because I have a unique ID field on all objects already. 

 

 

What my goal to do is load a MapInfo .TAB file into FME, find all gemoetric matches, and only keep one of each set of matched objects. For example, if I have 35 of the same line segment, I only want to retain 1 of that line segment.

 

 

Any next step guidance would be awesome.  Thanks!
takashi
Rank
Userlevel 2
Badge +17
Hi Nick,

 

 

Both of the Matcher and DuplicateRemover are transformers. There are many documentations in FMEpedia, I think "FME Getting Started Guide" and "FME Workbench" are suitable to begin FME Workbench. You can learn general usage of transformers in FME Workbench > Transformer.

 

 

> For example, if I have 35 of the same line segment, I only want to retain 1 of that line segment.

 

 

To do this, the Matcher can be used like this for example:

 

 

Takashi
N
Hi Guys,

 

 

Sorry for taking a while to get back to you; however, I was out of the office on vacation.  So I have the following workspace created to remove duplicates. I tested it out with some sample duplicate object data I created (point, line, polygon objects) are what we deal with here.

 

 

This gives me any object from the source that is geometrically unique, and for lack of terms "flattens" all duplicates based on geometry.

 

 

The problem I have is that in my source data I a schema with 25 columns and I want to maintain all of this column data "exactly" as it is in the source.

 

 

Is this possible to carry this data over "as is" into the final exported file that no long has duplicates?  With my beginner level status, I am guessing I would have to possible add a data inspector somewhere?

 

 

 

 

Thanks for any help in advance.

 

Nick

Reply


Community Stats

28,753
Posts
109,620
Replies
36,587
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.