Question

Dissolver running out of memory

  • 6 November 2019
  • 8 replies
  • 5 views
J
Badge

I have just updated some older workspaces that use dissolver transformes. Hoping for a performance boost, I upgraded all dissolvers to the current version (within FME 2019.1.3).

For some reason, this has lead to the workspaces crashing. The error message being displayed is "Insufficient memory available". I am trying to dissolve a dataset of ~8.000 features read from a shapefile which should yield ~100 features after dissolving. Looking at the windows task manager I can see the fme process using all my physical memory (16GB). The swap file generated by windows is about 16gb as well.

Before upgrading the transformers, this workspace used to run without any issues in FME 2018.X.

Since the features are ordered, I have also tried using group by mode "process when group changes" but the result ist the same.

Do you have any ideas on how to work around this?

8 replies

david_r
Rank
Userlevel 4

How are the tolerance settings configured? Have you tried deactivating the tolerance?

sigtill
Rank
Badge +21

I usually add a GeometryValidator before to verify all geometry. If I still have issues I use a 2DForcer if the 3D-dimension is not important. If that does not help I use the Tiler to tile the data first. If you have huge polygons with millions of points it sometimes helps to split them - even just into 2 parts.

The real SigTill
J
Badge

Geometries were validated and fixed outside of fme beforehand. Just to make sure, I tried your suggestion using the validator, but the dissolver still runs out of memory.

Tolerance is set to none.
ebygomm
Rank
Userlevel 1
Badge +10

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

J
Badge

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

david_r
Rank
Userlevel 4

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

J
Badge

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

fmelizard
Rank
Userlevel 4
Badge +13

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

Definitely will be interested in what the conclusion here ends up being -- very glad that it works with a rebuild, but it should have worked with just an update so we'll be very curious... please keep us posted.

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.