Skip to main content
Question

Dissolver running out of memory

  • November 6, 2019
  • 8 replies
  • 56 views
J
Forum|alt.badge.img

I have just updated some older workspaces that use dissolver transformes. Hoping for a performance boost, I upgraded all dissolvers to the current version (within FME 2019.1.3).

For some reason, this has lead to the workspaces crashing. The error message being displayed is "Insufficient memory available". I am trying to dissolve a dataset of ~8.000 features read from a shapefile which should yield ~100 features after dissolving. Looking at the windows task manager I can see the fme process using all my physical memory (16GB). The swap file generated by windows is about 16gb as well.

Before upgrading the transformers, this workspace used to run without any issues in FME 2018.X.

Since the features are ordered, I have also tried using group by mode "process when group changes" but the result ist the same.

Do you have any ideas on how to work around this?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

8 replies

david_r
Celebrity
  • david_rCelebrity
  • 8394 replies
  • November 6, 2019

How are the tolerance settings configured? Have you tried deactivating the tolerance?

sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • 956 replies
  • November 6, 2019

I usually add a GeometryValidator before to verify all geometry. If I still have issues I use a 2DForcer if the 3D-dimension is not important. If that does not help I use the Tiler to tile the data first. If you have huge polygons with millions of points it sometimes helps to split them - even just into 2 parts.

The real SigTill
J
Forum|alt.badge.img
  • jakob
  • Author
  • 12 replies
  • November 6, 2019

Geometries were validated and fixed outside of fme beforehand. Just to make sure, I tried your suggestion using the validator, but the dissolver still runs out of memory.

Tolerance is set to none.
ebygomm
Influencer
Forum|alt.badge.img+44
  • ebygommInfluencer
  • Influencer
  • 3427 replies
  • November 6, 2019

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

J
Forum|alt.badge.img
  • jakob
  • Author
  • 12 replies
  • November 6, 2019

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

david_r
Celebrity
  • david_rCelebrity
  • 8394 replies
  • November 6, 2019

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

J
Forum|alt.badge.img
  • jakob
  • Author
  • 12 replies
  • November 6, 2019

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

fmelizard
Safer
Forum|alt.badge.img+20
  • fmelizardSafer
  • Safer
  • 3719 replies
  • November 6, 2019

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

Definitely will be interested in what the conclusion here ends up being -- very glad that it works with a rebuild, but it should have worked with just an update so we'll be very curious... please keep us posted.

Community Stats

32,079
Posts
121,810
Replies
39,714
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.