Skip to main content
Question

Dissolver running out of memory

J
Forum|alt.badge.img

I have just updated some older workspaces that use dissolver transformes. Hoping for a performance boost, I upgraded all dissolvers to the current version (within FME 2019.1.3).

For some reason, this has lead to the workspaces crashing. The error message being displayed is "Insufficient memory available". I am trying to dissolve a dataset of ~8.000 features read from a shapefile which should yield ~100 features after dissolving. Looking at the windows task manager I can see the fme process using all my physical memory (16GB). The swap file generated by windows is about 16gb as well.

Before upgrading the transformers, this workspace used to run without any issues in FME 2018.X.

Since the features are ordered, I have also tried using group by mode "process when group changes" but the result ist the same.

Do you have any ideas on how to work around this?

8 replies

david_r
Celebrity
  • david_rCelebrity
  • November 6, 2019

How are the tolerance settings configured? Have you tried deactivating the tolerance?

sigtill
Supporter
Forum|alt.badge.img+24
  • sigtillSupporter
  • Supporter
  • November 6, 2019

I usually add a GeometryValidator before to verify all geometry. If I still have issues I use a 2DForcer if the 3D-dimension is not important. If that does not help I use the Tiler to tile the data first. If you have huge polygons with millions of points it sometimes helps to split them - even just into 2 parts.

The real SigTill
J
Forum|alt.badge.img
  • jakob
  • Author
  • November 6, 2019

Geometries were validated and fixed outside of fme beforehand. Just to make sure, I tried your suggestion using the validator, but the dissolver still runs out of memory.

Tolerance is set to none.
ebygomm
Influencer
Forum|alt.badge.img+38
  • ebygommInfluencer
  • Influencer
  • November 6, 2019

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

J
Forum|alt.badge.img
  • jakob
  • Author
  • November 6, 2019
ebygomm wrote:

Put the old dissolver back in? What version is the dissolver in 2019.1.3?

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

david_r
Celebrity
  • david_rCelebrity
  • November 6, 2019
jakob wrote:

I guess that might be a temporary solution to my problem. However, the dissolver in the 2019 version seems to perform a lot better in all other regards so I don't want to keep deprecated versions in my workspaces.

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

J
Forum|alt.badge.img
  • jakob
  • Author
  • November 6, 2019
david_r wrote:

Sounds like you may want to notify Safe support about this. They've recently rewritten the Dissolver (from scratch, I believe) and I'm certain they'd want to know about these things.

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • November 6, 2019
jakob wrote:

I'll do some more testing first. I have just rebuild the workspace from scratch with the excact same settings and it seems to work now. Maybe there was something wrong with the old file - I just upgraded the transformers within the workspace.

Definitely will be interested in what the conclusion here ends up being -- very glad that it works with a rebuild, but it should have worked with just an update so we'll be very curious... please keep us posted.

Reply


Helpful Members This Week

Community Stats

31,335
Posts
118,867
Replies
38,957
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.