Question

Missing Polygons

J
Badge

Hi There, I am in process of troubleshooting a workspace (2016 64 bit). The workspace sources data from oracle database and then matches the keys with few spatial layers followed by clean up and merging based on key attributes. The output of this process is a shapefile (approx. 100K records). I noticed that FME run misses some polygons that it should have included in the output. I ran the process with just the problem area and it works fine. The problem shows up when I run the full process. I am curious if someone has come across this issue and had any suggestions for me.

Thanks

Jay

11 replies

redgeographics
Rank
Userlevel 5
Badge +25

It's really hard to troubleshoot something like this without seeing the data. Given the size of your source dataset it's probably best to file this as a support case with either Safe or your reseller.

M
Badge +2

Hi @jaygis how about running in Full Inspection mode to see where features are being dropped, or is it at the writer stage? Any warnings in the log file?

M
Badge +2

Hi @jaygis;

 

I believe that you are joining the spatial data with some existing database, could you please confirm which transformer you are using and also the properties?

 

 

Thanks.

 

Lyes

 

J
Badge

Thanks All for your suggestions. I am going through more steps of identifying where the the problem is being generated. Lyes, this workspace is using Feature merger. I should point out that spatial layer that it uses for matching has 4.15M records (sourced from SDE). I am not sure if this is a factor. Thanks Jay

M
Badge +2

Thanks All for your suggestions. I am going through more steps of identifying where the the problem is being generated. Lyes, this workspace is using Feature merger. I should point out that spatial layer that it uses for matching has 4.15M records (sourced from SDE). I am not sure if this is a factor. Thanks Jay

Would you be able to identify the all the missing polygons using the ChangeDetector? By this way you should be able to look at the missing ones.

jneujens
Badge

Just a general remark jaygis: when reading from a database based on other features, a DatabaseJoiner will work faster than a Feature Merger. In your case (4.15M records) it might be helpful?

J
Badge

Thanks jneujens for your feedback, I am i process of upgrading FME to version 2017.1 and see if that provides any better results. if not, will try spliting the workspace so that it processes less data and also as well implement DatabaseJoiner. Thanks Jay

 

M
Badge +2

Thanks jneujens for your feedback, I am i process of upgrading FME to version 2017.1 and see if that provides any better results. if not, will try spliting the workspace so that it processes less data and also as well implement DatabaseJoiner. Thanks Jay

 

Would you be able to share your workspace with us?

 

 

G
Badge +3

R you using a featuremerger with no group by?

If the objects dropped have correct matching value's for the merge field(s), then maybe it is an encoding problem of the mergeattribute(s).

I encountered this (only once to), part of the merge attribute had different encoding and did not merge.

I used tcl code to recode the objects.

J
Badge

Hi Geo, Thanks for this pointer. Feature merger doesn't use group by. I am currently generating intermediate outputs so that way I can see which part of workspace gets impacted with full run. Thanks

takashi
Rank
Userlevel 2
Badge +17

Hi @jaygis, then, were you able to specify the location where the polygons have been lost in your workspace? If the polygons were lost at the FeatureMerger, check if the lost features have been output via the UnmergedRequestor (NotMerged) port.

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.