Skip to main content
Question

Aggregator or FGDB Writer change in behaviour between version 11 and 13?

  • October 16, 2017
  • 8 replies
  • 44 views
tim_wood
Contributor
Forum|alt.badge.img+8

Is there a change in

I'm re-creating a Workspace in 64-bit Desktop 2017.1.1.0 (17650). The Workspace takes a ShapeFile of line data and process it in varies ways, including Aggregating on 3 fields - Geometry - Assemble One Level, Drop Incoming Attributes, Homogenous Collection (If Possible) - and writing to a File Geodatabase.

The old Workspace included an Aggregator transformer version 11 and works in FME 2016.1. However in 2017.1.1 with Aggregator version 14 I get 35 fewer features in the File Geodatabase. The rejected features are logged like this:

2017-10-16 12:39:14| 92.3| 0.1|WARN |FileGDB Writer: Failed to write Geometry to feature class 'roads_classnumber' with geometry type 'esriGeometryPolyline'. Dropping containing feature

 

2017-10-16 12:39:14| 92.3| 0.0|STATS |Storing feature(s) to FME feature store file `X:\\x_log.ffs'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Feature Type: `roads_classnumber'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `class_number' has value `B365'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `filegdb_type' has value `geodb_polyline'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `fme_geometry' has value `fme_aggregate'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `fme_type' has value `fme_line'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `multi_writer_id' has value `0'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `posttown' has value `WALTON-ON-THAMES'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `village' has value `'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Coordinate System: `_OSGB-GPS-2015_0'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Type: IFMEAggregate

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Front Appearance Reference: `<inherited_or_default_appearance>'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Back Appearance Reference: `<inherited_or_default_appearance>'

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Number of Geometries: 42

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 0

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 3 -- Coordinate Dimension: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510242,166083)(510232,166072)(510222,166059)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 1

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 2 -- Coordinate Dimension: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510222,166059)(510221,166092)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 8 -- Coordinate Dimension: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510106,165657)(510100,165634)(510083,165572)(510063,165500)(510041,165412)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510039,165405)(510026,165355)(510025,165354)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 3

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 4 -- Coordinate Dimension: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510176,165937)(510169,165912)(510161,165883)(510157,165865)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 4

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 6 -- Coordinate Dimension: 2

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | ...Skipping coordinates...

 

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510123,165713)

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |Skipping parts...

 

2017-10-16 12:39:14| 92.3| 0.0|WARN |===========================================================================
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

8 replies

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • October 16, 2017

OK I've found a reason - the old Workspace was using the File Geodatabase (Geodb - used to be ArcObjects) Writer i.e. the one that uses ArcGIS Desktop to write the FGDB. The new Workspace was using the File Geodatabase Open API Writer. I vaguely remember some comment about that from the past. Does the Open API Writer not support multi-part features?

mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • October 17, 2017

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

FME Evangelist to the Rich and Famous!!
tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • October 18, 2017

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

Hi @Mark2AtSafe thanks for your comments.

 

The Workspace takes a ShapeFile of road network centrelines where each line is a junctino to junction section. Various outputs are produced including aggregating on unique street reference and creating a point layer to use as a gazetteer. In answer to your questions...

 

1) I'll double check but it's possible as some outputs are based on other outputs with additional processing. I presume you mean a multi-part feature that contains multi-part features, to use ESRI terminology...?

 

2) It's just 2D lines. There's no intentional front and back appearance in the data. Would something like this come in with GeometryValidator (all checks and repairs selected) and/or GeometryFilter? GeometryFilter comes before most of the processing and I'm using only what comes out of the Line port. Everything else goes to an Inspector.

 

3) All components should just be 2D lines, unless aggregates of aggregates count as different geometry types.

 

david_r
Celebrity
  • david_rCelebrity
  • October 18, 2017

OK I've found a reason - the old Workspace was using the File Geodatabase (Geodb - used to be ArcObjects) Writer i.e. the one that uses ArcGIS Desktop to write the FGDB. The new Workspace was using the File Geodatabase Open API Writer. I vaguely remember some comment about that from the past. Does the Open API Writer not support multi-part features?

Aggregates are supported in the File Geodb API writer as well, see: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/filegdb/quick_facts_filegdb.htm

 

 

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • October 18, 2017

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

@Mark2AtSafe

 

There's only Aggregator before the Writer. I'll email my Workspace and source data to Support.

 

For info, I've tried the 2016 Workspace with a File Geodatabase API Writer and get the same problem - 35 features are rejected as above.
nic_ran
Contributor
Forum|alt.badge.img+17
  • nic_ranContributor
  • Contributor
  • April 4, 2019

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • April 29, 2019

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

I'm taking a look right now. I confirm the behaviour but I'm checking in with the developers to see if this is expected or not. I guess it might not be supported by the api, though if it can read nested aggregates then I would expect writing to work too.

FME Evangelist to the Rich and Famous!!
mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • April 29, 2019

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

And very quickly I find that there is an existing issue. fyi the reference number is FMEENGINE-10103. The developer comment there is "Repro'd. We should be able to handle arbitrary nesting for homogenous aggregates." - so definitely a shortcoming in the writer.

FME Evangelist to the Rich and Famous!!

Community Stats

32,413
Posts
122,925
Replies
40,132
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.