Skip to main content

Hi,

I am reading in a DGN TIN and then using a GeometryCoercer transformer to convert the data to fme_polygon. I am then outputting the polygons to a Shapefile writer.

However the incoming dataset is so large that it is failing to write out completely to shapefile due to shapefile's size limit of 2GB.

Is there any transformer I can use that will somehow compress the data so that I can write it to shapefile (I need shapefile and can't use any other output format)?

I am using FME 2017.1 64 Bit

Thank you in advance.

Are there any attributes that you can remove before output?

 

 


Hi, I have removed all attributes already and unfortunately it is still too big,,,

 


You could possibly try using a Generalizer to reduce the density of points on each polygon. You would just need to test to make sure the resulting geometry is close enough to the original to be acceptable.


Hi @aquamarine,

You could try using the TINGenerator to rebuild the TIN surface from the DGN input, then writing the output TINSurface feature to Shape as a multipatch. This will be more efficient than individual polygons, and keeps the original intent of the DGN data.

This will be a memory-intensive process, so I recommend running it on a 64 bit version of FME.


Some attribute data types are more 'expensive' than others, so consider adjusting width and rounding values to fewer decimal places if necessary too if necessary. There's also the ability to compress the file to a .shz (in the writer parameters), but that assumes you can of course close the writer in the first place.

Finally, do you need the full extents? Perhaps trim the geography with the Clipper to lose a little extra weight.

Is the DBF the heavy one or the SHP? That'll give you a clue where you need to trim or you could try to check again if Shape truly has to be the target format! There seems to be a movement out there slating the humble yet ubiquitous format!


Some attribute data types are more 'expensive' than others, so consider adjusting width and rounding values to fewer decimal places if necessary too if necessary. There's also the ability to compress the file to a .shz (in the writer parameters), but that assumes you can of course close the writer in the first place.

Finally, do you need the full extents? Perhaps trim the geography with the Clipper to lose a little extra weight.

Is the DBF the heavy one or the SHP? That'll give you a clue where you need to trim or you could try to check again if Shape truly has to be the target format! There seems to be a movement out there slating the humble yet ubiquitous format!

Thank you @1spatialdave .

 

The DBF is relatively small (2.8MB) compared to the SHP file which is 2.8GB.

 

 

I tried rounding the values to fewer decimal places and then compressing it to a .shz file. But now I can't work out how to uncompress this file for use in ArcGIS....

 

 


I don't think there is a way to save the data larger than 2GB with the Shapefile format, unless you divide the dataset into some small sub datasets or reduce the contents of the dataset.

If the TIN surface is a terrain model, the TINGenerator with setting some tolerance could be used to simplify the surface (i.e. reduce the contents) as @DaveAtSafe suggested.


Thank you @1spatialdave .

 

The DBF is relatively small (2.8MB) compared to the SHP file which is 2.8GB.

 

 

I tried rounding the values to fewer decimal places and then compressing it to a .shz file. But now I can't work out how to uncompress this file for use in ArcGIS....

 

 

Yes, .shz is not understood by ArcGIS (only by FME). So that step won't help you. Sounds like having less geometry is the way forward. However, I feel like that may be the same as telling Mozart you like his music but it would be better with fewer notes. If it is a TIN you've got, perhaps put it in a TINGenerator and create a new TIN with a higher surface tolerance -- that will reduce the # of polygons and therefore make your file smaller...but with less accuracy.

 

 


Reply