Skip to main content

Hello!

 

I am working at the moment on several FME Workspaces that can process data with as little manual manipulation as possible. In general just a file is selected, and from the data and its schema, automatically whole schema's in databases are created, filled, and further processed for further use. Obviously dynamic readers and writers are used.

 

Now I have a Challenge to recognize existing objects based on spatial relations, new data and existing data from a central database. There are several approaches here:

 

1. to compare geometries on basis of coordinates. A SpatialFilter could give for example possibilities there;

2. to do a fuzzy selection under the asssumptions that (point) objects in the field survey have been recognized but may not have exactly the same position. A NeighborFinder could here for example give solutions there.

 

In an inital approach I tried to use the SpatialFilter and NeighborFinder and to use the "Group by"-functionality using the attribute fme_feature_type.

 

Unfortunately, when you do that (using FME Workbench 2020.0) the output given by the Transformers of SpatialFilter and NeighborFinder do not give the expected outcome. All data will have an outcome of "Failed" for the SpatialFilter and "UnmatchedBase" for the NeighborhoodFinder. (I suspect that fme_feature_type is used within the Transformers itself and therefore can not be used twice.)

 

Does anyone have ideas to come to elegant solutions here?

 

Best regards, Jochgem

The fme_feature_type can be used as a group by in Spatial Filter/Neighbourfinder so the problem probably lies elsewhere


The fme_feature_type can be used as a group by in Spatial Filter/Neighbourfinder so the problem probably lies elsewhere

HI there!

 

Thanks for your answer.

 

Did you base this statement on a test using dynamic readers and writers?

 

Best, Jochgem


HI there!

 

Thanks for your answer.

 

Did you base this statement on a test using dynamic readers and writers?

 

Best, Jochgem

There can be some issues with the feature attributes being exposed correctly with featurereaders depending on file type. Have you verified that the fme_feature_type is populated?


There can be some issues with the feature attributes being exposed correctly with featurereaders depending on file type. Have you verified that the fme_feature_type is populated?

Hi there!

 

I double checked and the fme_feature_type is populated.

 

I do have to rephrase the problem as I noticed that the SpatialFilter and NeighborhoodFinder DO have an outcome but do not have the results you would expect. For the SpatialFilter the outcome is for all cases "Failed" while for the NeighborhoodFinder all the outcomes are "Unmatched Base" (and "Unmatched Candidate"). As the data comes for most cases from the same original source (one being the original source while the other populating a new database) I would expect different results.


There can be some issues with the feature attributes being exposed correctly with featurereaders depending on file type. Have you verified that the fme_feature_type is populated?

Could it be that, while viewing the data in the FME Data Inspector, that the value for the fme_feature_type looks the same but is actually different within the Transformer (and with that hidden for the user), resulting in separate groups within the Transformer?

Firstly make sure that the Filter features and the Candidate features which you are going to compare have the same value in the fme_feature_type attribute.


Firstly make sure that the Filter features and the Candidate features which you are going to compare have the same value in the fme_feature_type attribute.

Dear Tagashi,

 

Thank you for your comment. It is already for years that I see you prominently present at the FME Knowledge Center. Great that you react also here on this post!

 

The fme_feature_type are exactly the same. I verified that multiple times.

 

My approach is as follows.

A File Geodatabase (the original data source) is used in two ways:

1. to populate a PostGIS/PostGreSQL-database.

2. to create test data. Hereby a few features have been manually manipulated.

 

Both data sources, from the PostGIS/PostGreSQL database and from the altered File Geodatabase, are compared to each other using dynamic readers and a SpatialFilter (or NeighborFinder) to make spatial comparisons.

 

If I set the equation form to "is equal to" in the SpatialFilter still all features go out from the Failed Port, while I would expect that the features that have not been manipulated in the test data would at least go out of the Passed Port.

 

Best, Jochgem


Dear Tagashi,

 

Thank you for your comment. It is already for years that I see you prominently present at the FME Knowledge Center. Great that you react also here on this post!

 

The fme_feature_type are exactly the same. I verified that multiple times.

 

My approach is as follows.

A File Geodatabase (the original data source) is used in two ways:

1. to populate a PostGIS/PostGreSQL-database.

2. to create test data. Hereby a few features have been manually manipulated.

 

Both data sources, from the PostGIS/PostGreSQL database and from the altered File Geodatabase, are compared to each other using dynamic readers and a SpatialFilter (or NeighborFinder) to make spatial comparisons.

 

If I set the equation form to "is equal to" in the SpatialFilter still all features go out from the Failed Port, while I would expect that the features that have not been manipulated in the test data would at least go out of the Passed Port.

 

Best, Jochgem

If you remove the group by for testing purposes, does anything pass?

I suspect there may be slight differences in geometries due to coordinate rounding or similar which often happens when you write to a filegdb - the spatial filter test "is equal to" has no tolerance as far as i recall. I'd look at a feature from each source that you would expect to match in the feature information window to see if the geometry differs


Firstly make sure that the Filter features and the Candidate features which you are going to compare have the same value in the fme_feature_type attribute.

Could you please share sample source datasets with us?

No need to share all features. Extract each 100 features, for example, from the Filter features and Candidate features (at least keep fme_feature_type and geometry as-is), save them into FFS - FME Feature Store file and upload it here, that's enough.


Could you please share sample source datasets with us?

No need to share all features. Extract each 100 features, for example, from the Filter features and Candidate features (at least keep fme_feature_type and geometry as-is), save them into FFS - FME Feature Store file and upload it here, that's enough.

Hi there!

 

Thanks (both) for your quick reply. I have to go now, but I will come back to you.

 

Best, Jochgem


@jochgem

Just a thought: are both datasets in the same coordinate system?

These transformers do not convert coordinate systems and that can lead to unmatched features.

Hope this helps.


Hi there!

 

Only today I found time again to analyze this problem.

 

Something interested happened. Dynamic readers can introduce the Published Parameter "Feature Types" in which you can select one feature type, multiple feature types or leave the field empty to select all feature types. This is what happened when I tested my FME Workspace. I did 3 tests:

 

1. I selected one feature type. From the PostGIS/PostGreSQL a fme_feature_type without schema name is given;

 

2. I select two (that is, multiple) feature types. From the PostGIS/PostGreSQL both fme_feature_type's without schema name are given;

 

3. I leave the "Feature Types" empty. In this case PostGIS/PostGreSQL all fme_feature_type's have values including the schema name. When a comparison is made on basis of grouping of fme_feature_types all output goes to the Failed Port as FME "sees" no similarity between the groups.

 

 


@jochgem

Just a thought: are both datasets in the same coordinate system?

These transformers do not convert coordinate systems and that can lead to unmatched features.

Hope this helps.

Thanks, Erik Jan!

 

I will take this into account in further analysis.


Reply