Aggregator or FGDB Writer change in behaviour between version 11 and 13? | Community

Is there a change in

I'm re-creating a Workspace in 64-bit Desktop 2017.1.1.0 (17650). The Workspace takes a ShapeFile of line data and process it in varies ways, including Aggregating on 3 fields - Geometry - Assemble One Level, Drop Incoming Attributes, Homogenous Collection (If Possible) - and writing to a File Geodatabase.

The old Workspace included an Aggregator transformer version 11 and works in FME 2016.1. However in 2017.1.1 with Aggregator version 14 I get 35 fewer features in the File Geodatabase. The rejected features are logged like this:

2017-10-16 12:39:14| 92.3| 0.1|WARN |FileGDB Writer: Failed to write Geometry to feature class 'roads_classnumber' with geometry type 'esriGeometryPolyline'. Dropping containing feature

2017-10-16 12:39:14| 92.3| 0.0|STATS |Storing feature(s) to FME feature store file `X:\\x_log.ffs'

2017-10-16 12:39:14| 92.3| 0.0|WARN |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2017-10-16 12:39:14| 92.3| 0.0|WARN |Feature Type: `roads_classnumber'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `class_number' has value `B365'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `filegdb_type' has value `geodb_polyline'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `fme_geometry' has value `fme_aggregate'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `fme_type' has value `fme_line'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(string) : `multi_writer_id' has value `0'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `posttown' has value `WALTON-ON-THAMES'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Attribute(encoded: utf-16): `village' has value `'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Coordinate System: `_OSGB-GPS-2015_0'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Type: IFMEAggregate

2017-10-16 12:39:14| 92.3| 0.0|WARN |Front Appearance Reference: `<inherited_or_default_appearance>'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Back Appearance Reference: `<inherited_or_default_appearance>'

2017-10-16 12:39:14| 92.3| 0.0|WARN |Number of Geometries: 42

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 0

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 3 -- Coordinate Dimension: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510242,166083)(510232,166072)(510222,166059)

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 1

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 2 -- Coordinate Dimension: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510222,166059)(510221,166092)

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 8 -- Coordinate Dimension: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510106,165657)(510100,165634)(510083,165572)(510063,165500)(510041,165412)

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510039,165405)(510026,165355)(510025,165354)

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 3

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 4 -- Coordinate Dimension: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510176,165937)(510169,165912)(510161,165883)(510157,165865)

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Geometry Number: 4

2017-10-16 12:39:14| 92.3| 0.0|WARN | Geometry Type: IFMELine

2017-10-16 12:39:14| 92.3| 0.0|WARN | Number of Coordinates: 6 -- Coordinate Dimension: 2

2017-10-16 12:39:14| 92.3| 0.0|WARN | ...Skipping coordinates...

2017-10-16 12:39:14| 92.3| 0.0|WARN | (510123,165713)

2017-10-16 12:39:14| 92.3| 0.0|WARN |--------------------------------------

2017-10-16 12:39:14| 92.3| 0.0|WARN |Skipping parts...

2017-10-16 12:39:14| 92.3| 0.0|WARN |===========================================================================

Page 1 / 1

OK I've found a reason - the old Workspace was using the File Geodatabase (Geodb - used to be ArcObjects) Writer i.e. the one that uses ArcGIS Desktop to write the FGDB. The new Workspace was using the File Geodatabase Open API Writer. I vaguely remember some comment about that from the past. Does the Open API Writer not support multi-part features?

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

Hi @Mark2AtSafe thanks for your comments.

The Workspace takes a ShapeFile of road network centrelines where each line is a junctino to junction section. Various outputs are produced including aggregating on unique street reference and creating a point layer to use as a gazetteer. In answer to your questions...

1) I'll double check but it's possible as some outputs are based on other outputs with additional processing. I presume you mean a multi-part feature that contains multi-part features, to use ESRI terminology...?

2) It's just 2D lines. There's no intentional front and back appearance in the data. Would something like this come in with GeometryValidator (all checks and repairs selected) and/or GeometryFilter? GeometryFilter comes before most of the processing and I'm using only what comes out of the Line port. Everything else goes to an Inspector.

3) All components should just be 2D lines, unless aggregates of aggregates count as different geometry types.

OK I've found a reason - the old Workspace was using the File Geodatabase (Geodb - used to be ArcObjects) Writer i.e. the one that uses ArcGIS Desktop to write the FGDB. The new Workspace was using the File Geodatabase Open API Writer. I vaguely remember some comment about that from the past. Does the Open API Writer not support multi-part features?

Aggregates are supported in the File Geodb API writer as well, see: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/filegdb/quick_facts_filegdb.htm

Actually the Readers and Writers manual says that aggregate features are supported by the open API version. I know you mentioned some of these in the question, but the only things I can think of to cause a difference are...

1) A nested aggregate - ie an aggregate of aggregates. That might be one reason for a difference. In that scenario maybe you can deaggregate to a single level of aggregation?

2) I notice you have a front and back appearance. Is the aggregate part of a surface? Surfaces aren't supported by the API writer.

3) Is it a homogeneous aggregate (ie all components are the same geometry)?

I don't know if any of that will help, but they're reasons I can think why an aggregate might not be supported. To be sure I think you'd have to send us a copy of the data - or at least a copy of the FFS spatial log file - to see what is happening. Or maybe post a few screenshots of the FFS log in the data inspector (its attributes and geometry in the feature info window).

@Mark2AtSafe

There's only Aggregator before the Writer. I'll email my Workspace and source data to Support.

For info, I've tried the 2016 Workspace with a File Geodatabase API Writer and get the same problem - 35 features are rejected as above.

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

I'm taking a look right now. I confirm the behaviour but I'm checking in with the developers to see if this is expected or not. I guess it might not be supported by the api, though if it can read nested aggregates then I would expect writing to work too.

I think we've tracked down the issue. It seems that the ArcObjects file GDB writer supports multi-level aggregates but the Open API version will only support writing single level aggregates. Both versions appears to be able to read multi-level aggregates.

Perhaps Safe can look into fixing this behaviour with the Open API File GDB writer (@mark2atsafe)?

And very quickly I find that there is an existing issue. fyi the reference number is FMEENGINE-10103. The developer comment there is "Repro'd. We should be able to handle arbitrary nesting for homogenous aggregates." - so definitely a shortcoming in the writer.

Reply