Solved

Loading a large CityGML dataset

  • 10 July 2018
  • 5 replies
  • 11 views

Badge

Hi,

Is there a more efficient way of loading a large CityGML dataset, approx. 15GB unzipped (http://www.3dcitydb.net/3dcitydb/fileadmin/mydata/Cesium_NYC_Demo/CityGML/NYC_Buildings_CityGML_20150907.zip) apart from using the standard reader.

I was thinking of using this approach (experimental?) but tests using a smaller dataset did not yield any advantage compared to the standard reader.Thanks
icon

Best answer by virtualcitymatt 12 July 2018, 11:22

View original

5 replies

Userlevel 4
Badge +26

I wouldn't really recommend loading the whooole data set into FME - this will just take to much memory. I think your best bet would be to use the 3dcitydb importer/exporter tool to import the citygml into a postgis instance of a 3D city database (this will take quite a bit of time too). Once in the database you can use the 3dcitydb importer/exporter tool export the data as tiles. These tiles I think could be what FME/Your hardware would be able to handle a little better.

 

 

Here is where you can get the importer/exporter: https://www.3dcitydb.org/3dcitydb/d3dimpexp/ also read the documentation on how to get it all working.
Badge

I wouldn't really recommend loading the whooole data set into FME - this will just take to much memory. I think your best bet would be to use the 3dcitydb importer/exporter tool to import the citygml into a postgis instance of a 3D city database (this will take quite a bit of time too). Once in the database you can use the 3dcitydb importer/exporter tool export the data as tiles. These tiles I think could be what FME/Your hardware would be able to handle a little better.

 

 

Here is where you can get the importer/exporter: https://www.3dcitydb.org/3dcitydb/d3dimpexp/ also read the documentation on how to get it all working.
@virtualcitymatt Thanks for the info, I'll give this a try.

 

Badge

Just to follow up with this thread - I have loaded the whole NYC LoD2 GML file into the 3D City Database, but am unsure what you mean by outputting tiles. I saw a filter function though, presumably I can output only the "building" classes.

 

A question is, how can I output individual buildings into individual files? I thought maybe I need to write a customized script to do this, but just in case you know any off-the-shelf tools. Thanks!

Badge

Just to follow up with this thread - I have loaded the whole NYC LoD2 GML file into the 3D City Database, but am unsure what you mean by outputting tiles. I saw a filter function though, presumably I can output only the "building" classes.

 

A question is, how can I output individual buildings into individual files? I thought maybe I need to write a customized script to do this, but just in case you know any off-the-shelf tools. Thanks!

You can use FME to export individual buildings to individual gml files from 3D City Database.

To do this you need xml project file to the city database. You can save one from the GUI of the 3dcityDB importer exporter (project at the top (before saving to a project file, establish your connection and do every other preference settings in the preference and export tabs)). Save this file to a location where you can easily find it.

Then in FME use SQL executor to select gmlid from your database as result. Modify the result value with attribute creator, so that the new value is <gmlid>(gml id value)</gmlid>. Now it's suitable to a xml document. Send this new gmlid to AttributeCreator where you have your whole, previously saved, 3DcityDB project xml copied as attribute value. Name the new attribute for example _xml.

The result gmlid from the SQLexecutor should be applied to the export filter part of the project xml in the attribute creator. <export>.....<filter>....<mode>...<simple>..<gmlIds>(put your gmlid value here)</gmlIds>. Thats why whe modified the gmlid to xml form.

Create a new project xml file with attribute file writer, this is your config file. Set parameter to owerwrite existing file. Now you have an configuration to export just one object defined by its gmlid.

With system caller in FME, trigger the 3D city Database export with cmd prompt commands and use your newly created project xml file as project xml file to be used in the export. With parameter and/or attribute values, name each output export file with f.e the gmlid value you resulted from sql executor. Now the export should be triggered for each building and each building should be in their own files. I recommend that you make user parameter value for the export path, and name the last folder with f.e the gmlid also, so that for each individual gml file there is also individual appearance folder (if you have appearances applied)

Example system caller command line:

java -jar -Xms128m -Xmx1024m (path to your 3d importer-exporter)/lib/3dcitydb-impexp.jar -shell -config (path to your config file) -export (your export path)/(your gml id value).gml

Badge

You can use FME to export individual buildings to individual gml files from 3D City Database.

To do this you need xml project file to the city database. You can save one from the GUI of the 3dcityDB importer exporter (project at the top (before saving to a project file, establish your connection and do every other preference settings in the preference and export tabs)). Save this file to a location where you can easily find it.

Then in FME use SQL executor to select gmlid from your database as result. Modify the result value with attribute creator, so that the new value is <gmlid>(gml id value)</gmlid>. Now it's suitable to a xml document. Send this new gmlid to AttributeCreator where you have your whole, previously saved, 3DcityDB project xml copied as attribute value. Name the new attribute for example _xml.

The result gmlid from the SQLexecutor should be applied to the export filter part of the project xml in the attribute creator. <export>.....<filter>....<mode>...<simple>..<gmlIds>(put your gmlid value here)</gmlIds>. Thats why whe modified the gmlid to xml form.

Create a new project xml file with attribute file writer, this is your config file. Set parameter to owerwrite existing file. Now you have an configuration to export just one object defined by its gmlid.

With system caller in FME, trigger the 3D city Database export with cmd prompt commands and use your newly created project xml file as project xml file to be used in the export. With parameter and/or attribute values, name each output export file with f.e the gmlid value you resulted from sql executor. Now the export should be triggered for each building and each building should be in their own files. I recommend that you make user parameter value for the export path, and name the last folder with f.e the gmlid also, so that for each individual gml file there is also individual appearance folder (if you have appearances applied)

Example system caller command line:

java -jar -Xms128m -Xmx1024m (path to your 3d importer-exporter)/lib/3dcitydb-impexp.jar -shell -config (path to your config file) -export (your export path)/(your gml id value).gml

Hi Jerez, sorry for the late reply and thanks so much for your help! I somehow just saw this. I eventually solved the issue by using the fanout function. For anyone else who might have stumbled upon this question, here is another thread I opened to resolve my problem: https://knowledge.safe.com/questions/84869/output-buildings-into-individual-obj-files.html

Reply