Question

Loss of data when writing CityGML features without cache

  • 16 February 2022
  • 6 replies
  • 2 views
denisgiannelli
Badge

Hello everybody,

 

For the sake of the question, my computer is a MacBook Pro (13-inch, Mid 2012) equipped with a 2.5Ghz Dual-Core Intel Core i5 processor, 8Gb 1600 Mhz DDR3 of memory and a 1Tb SSD.

 

I am authoring a workspace from NONE to CITYGML.

  • It uses some HTTPCaller transformers to get datasets from a WFS, based on locational user attributes (lat long of the center point and cardinal offsets for the model extents)
  • It models LoD2 trees, LoD2 buildings and LoD1 relief in compliance with the GML 2.0 schema, including appearance and X3D materials.
  • It finally writes all features in a CityModel, which is composed of several feature types on FME (CityModel, Building, Building Parts, etc...)
  • image

 

The workspace runs without any errors when feature caching is enabled, and it delivers all features validated according to the CityGML schema. Examples:

  • Center point -23.593186, -46.669269, offset 50m to all cardinal directions.image
  • Same center point, offset 100m to all cardinal directions.image

 

The problem rises when I turn feature caching off and try to run this workbench at once, i.e. from None to CityGML. Unexpected errors appear regarding xml validation, as if my model was not in compliance with CityGML.

  • image

Because of the fact that it is the same workspace and same user parameters as before, I am sure that this is not the problem. So I get the impression that, when running the workspace at once, FME Desktop "goes faster that it can", and therefore some important data are skipped in the flow. I have read about breakpoints, but if I am not mistaken this should be used for developing process, not for final products.

 

Any ideas on how I could make the workspace "wait" right before every CityGML writer feature type, and only then start writing the final file?

The number of features at every feature type writer varies according to the user parameters. The idea is publishing this workspace on FMEServer so that it could be used in an app that delivers the model according to the user's choices.

 

Thanks in advance!

Greetings,

Denis

6 replies

hkingsbury
Rank
Userlevel 5
Badge +29

I'm going to remove the GML aspect from my answer as i've never used it or really know much about it - @virtualcitymatt​ can probably help out more with GML specifics.

 

From what I can see (without knowing the intricacies of your process) I think you might be encountering a bit of an order of operations confusion. eg you're wanting things to run in the order 'Ar' 'Bw' 'Cr' 'Dw' (lower case r meanign read, w meaning write), but instead you're probably getting 'Ar' 'Cr' 'Bw' 'Dw'.

 

To solve this i'd look at using feature readers and writers. This will allow you to perform 'Ar' and then 'Bw'. The output of the 'Bw' feature writer can then be used to trigger 'Cr' and subsequently 'Dw'

 

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Transformers/featurereader.htm

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Transformers/featurewriter.htm

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd
virtualcitymatt
Rank
Userlevel 4
Badge +26

Are you writing out multiple CityGML files? I've seen FME crash/ERROR when writing out multiple files of CityGML. I haven't been able to pinpoint exactly where the issue comes from as not all processes seem to be affected. Perhaps it is related to the order of operations.

 

The next thing to test is to turn of the validation step to see if it has any effect.

denisgiannelli
Badge

Are you writing out multiple CityGML files? I've seen FME crash/ERROR when writing out multiple files of CityGML. I haven't been able to pinpoint exactly where the issue comes from as not all processes seem to be affected. Perhaps it is related to the order of operations.

 

The next thing to test is to turn of the validation step to see if it has any effect.

Hi @virtualcitymatt​ ,

I am writing only one CityGML file, with multiple feature types (CityModel, Building, BuildingPart, .....).

 

I installed FME Desktop on another computer to test if the computer was the issue. The result is however the same.

 

Then I followed your piece of advice and removed the XML validation step. In turns out that the workspace runs without fatal error, and the log file points that all features were written successfully.

image 

But when I inspect the CityGML file, I see all sorts of results.

 

This last time, for instance, FMEDesktop only wrote the SolitaryVegetationObject features and two X3D Materials. All other feature types weren't written.

image 

On another test, it wrote all cityobjectmember features, but did not write the x3D materials

image 

On another test, it wrote everything but the trees.

image 

Notice that I haven't changed anything in terms of input parameters, I am just trying to run the same workspace again and again.

 

It seems that I need to ask FME to only start writing feature types once all processes were completed. Only then, with the available memory, FMEDesktop could start writing the output file.

denisgiannelli
Badge

I'm going to remove the GML aspect from my answer as i've never used it or really know much about it - @virtualcitymatt​ can probably help out more with GML specifics.

 

From what I can see (without knowing the intricacies of your process) I think you might be encountering a bit of an order of operations confusion. eg you're wanting things to run in the order 'Ar' 'Bw' 'Cr' 'Dw' (lower case r meanign read, w meaning write), but instead you're probably getting 'Ar' 'Cr' 'Bw' 'Dw'.

 

To solve this i'd look at using feature readers and writers. This will allow you to perform 'Ar' and then 'Bw'. The output of the 'Bw' feature writer can then be used to trigger 'Cr' and subsequently 'Dw'

 

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Transformers/featurereader.htm

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Transformers/featurewriter.htm

Hi @hkingsbury,

 

I will have a look at these two transformers. Thanks!

virtualcitymatt
Rank
Userlevel 4
Badge +26

Hi @virtualcitymatt​ ,

I am writing only one CityGML file, with multiple feature types (CityModel, Building, BuildingPart, .....).

 

I installed FME Desktop on another computer to test if the computer was the issue. The result is however the same.

 

Then I followed your piece of advice and removed the XML validation step. In turns out that the workspace runs without fatal error, and the log file points that all features were written successfully.

image 

But when I inspect the CityGML file, I see all sorts of results.

 

This last time, for instance, FMEDesktop only wrote the SolitaryVegetationObject features and two X3D Materials. All other feature types weren't written.

image 

On another test, it wrote all cityobjectmember features, but did not write the x3D materials

image 

On another test, it wrote everything but the trees.

image 

Notice that I haven't changed anything in terms of input parameters, I am just trying to run the same workspace again and again.

 

It seems that I need to ask FME to only start writing feature types once all processes were completed. Only then, with the available memory, FMEDesktop could start writing the output file.

ok that's super weird I've never seen that before - can you try removing/disabling the CityModel feature type in the output. FME will just create this by default anyway and it could somehow be related to this.

 

Another thing I would look at is the Appearence and X3DMMaterial feature types - If the features themselves have the appearences then FME will also automatically create these as well - unless of course there is a specific reason you are defining these outputs.

denisgiannelli
Badge

Hi @virtualcitymatt​ ,

I am writing only one CityGML file, with multiple feature types (CityModel, Building, BuildingPart, .....).

 

I installed FME Desktop on another computer to test if the computer was the issue. The result is however the same.

 

Then I followed your piece of advice and removed the XML validation step. In turns out that the workspace runs without fatal error, and the log file points that all features were written successfully.

image 

But when I inspect the CityGML file, I see all sorts of results.

 

This last time, for instance, FMEDesktop only wrote the SolitaryVegetationObject features and two X3D Materials. All other feature types weren't written.

image 

On another test, it wrote all cityobjectmember features, but did not write the x3D materials

image 

On another test, it wrote everything but the trees.

image 

Notice that I haven't changed anything in terms of input parameters, I am just trying to run the same workspace again and again.

 

It seems that I need to ask FME to only start writing feature types once all processes were completed. Only then, with the available memory, FMEDesktop could start writing the output file.

Hi @virtualcitymatt,

In the end, it turns out that the "fanout dataset" option was on.

Since the name of my output CityGML file is based on a timestamp, and each output feature type is reached at different moments in time, the city model features were being written in multiple files with different names (because of different timestamps at every minute that passes during the translation).

A silly mistake, unfortunately. The workaround was to create a script user parameter that defines a unique timestamp at the beginning of the translation, and from there to assign the output file name based on this unique timestamp. In this sense, all features are now written in the same output file.

  • Side note 1: Removing/disabling the CityModel feature type would remove all attributes at CityModel level, and that's something I don't want to.
  • Side note 2: The cityobject features (walls, grounds, roofs, solitaryvegetationobjects, relief, etc.) do not have appearances by themselves... I am using the Appearance and X3DMaterial feature types to encode lists of Xlink references to their geometrical gml_id's.

Thanks for the hints anyway. Much appreciated!

Denis

Reply


Community Stats

28,714
Posts
109,490
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.