Question

User Feedback Request: Dissolver Performance


Userlevel 3
Badge +13

Dear FME Users,

 

 

Have you ever used Dissolver transformer? Did it serve the purpose?

 

 

Our Geometry Team is working on Dissolver Performance Improvement project. They have lots of brilliant ideas - while I do not understand all the details, the team is so excited I have no doubts Dissolver is up to remarkable improvement.

 

 

The Team would like to hear from the users. If you have a workflow that involves Dissolver and are not satisfied with the performance, please let us know. If you have a dataset Dissolver is struggling to dissolve, please share the dataset with us - we would like to make sure we test all the edge cases. If you have a dataset Dissolver is very efficient with, please don't hesitate to mention it as well - we want the updated Dissolver to be as fast as the current version or faster while processing your dataset.

 

 

Please add a comment below, share your dataset through Report a Problem, or drop us a line at dissolver@safe.com.

 

 

Thanks,

 

FME Lizard


7 replies

Badge +9

We use it on mass data (cadastral parcels). 9 Mio parcels read from GeoPackage dissolved

with Auto tolerance into 3000 aggregated parts. Total 12h with FME_TEMP 64Gb in one stash file (32-Bit FME 2018.0.1).

EDIT: Remaining holes caused by unwillingly deaggregated aggregates ;-)

Michael

Badge +9

I tried to use it several years ago to aggregate asset information for tax purposes. Using fire, amb, school districts, County boundary, and sections. I tried to get assets dissolved with each unique grouping. I was unable to accomplish what I needed. It became a multi step process in gis software.

Brad

Badge

We use it on mass data (cadastral parcels). 9 Mio parcels read from GeoPackage dissolved

with Auto tolerance into 3000 aggregated parts. Total 12h with FME_TEMP 64Gb in one stash file (32-Bit FME 2018.0.1).

EDIT: Remaining holes caused by unwillingly deaggregated aggregates ;-)

Michael

Hi @mhab

 

you have a good size dataset! Would you like us to test the updated Dissolver with it? This would provide a valuable comparison.

 

In general: were you satisfied with Dissolver performance in this case? Or did you maybe expect it to be x times faster? What were your expectations based on? Were you OK with unwillingly deaggregated aggregates? Or did expect the Dissolver to be smarter?

 

Badge

I tried to use it several years ago to aggregate asset information for tax purposes. Using fire, amb, school districts, County boundary, and sections. I tried to get assets dissolved with each unique grouping. I was unable to accomplish what I needed. It became a multi step process in gis software.

Brad

Hi @gisbradokla

 

were you not able to solve the problem using Dissolver because of Dissolver's performance or functionality limitations? Or was it simply easier to set up a multi step process with a GIS software then an FME workflow with Dissolver?

 

Badge +9
Hi @mhab

 

you have a good size dataset! Would you like us to test the updated Dissolver with it? This would provide a valuable comparison.

 

In general: were you satisfied with Dissolver performance in this case? Or did you maybe expect it to be x times faster? What were your expectations based on? Were you OK with unwillingly deaggregated aggregates? Or did expect the Dissolver to be smarter?

 

I think the default to deaggregate aggregates is good. In our case it needs some more testing if NOT deaggregating really solves the issue which I think it should.

 

 

EDIT: I was thinking in wrong ways. Not deaggregating is not an option and doesn't make any sense in our case. Existing options are: deaggregate or reject. As for now, this seems useful.

 

 

The testing with our data would not be that easy. The GPKG is 860Gb in size with the features of the dissolving case being only one table out of many.

 

 

At the moment I can't prepare an FFS sample short term as we are running other tests which should go undisturbed.

 

 

If you want I could try next week. Should that go via support ?

 

 

 

Badge
I think the default to deaggregate aggregates is good. In our case it needs some more testing if NOT deaggregating really solves the issue which I think it should.

 

 

EDIT: I was thinking in wrong ways. Not deaggregating is not an option and doesn't make any sense in our case. Existing options are: deaggregate or reject. As for now, this seems useful.

 

 

The testing with our data would not be that easy. The GPKG is 860Gb in size with the features of the dissolving case being only one table out of many.

 

 

At the moment I can't prepare an FFS sample short term as we are running other tests which should go undisturbed.

 

 

If you want I could try next week. Should that go via support ?

 

 

 

Yes, sure, via support (Report a problem form). Or you could send a download link to dissolver@safe.com. Next week is absolutely fine. The project is in development right now. We have another 2 or 3 months till completion. Getting some tricky data to test with any time between now and completion of the project would be very helpful.

 

Thank you in advance for your input.

 

 

Badge +9
Yes, sure, via support (Report a problem form). Or you could send a download link to dissolver@safe.com. Next week is absolutely fine. The project is in development right now. We have another 2 or 3 months till completion. Getting some tricky data to test with any time between now and completion of the project would be very helpful.

 

Thank you in advance for your input.

 

 

I did some testing with the large dataset with FME 2017 (32-bit) and FME 2018 (32-bit) and found considerable slower performance in 2018 (200+%)

 

 

I reported a problem and sent workspaces, Logs, download link for data and a summary.

 

 

I sure hope the underlying reason can be found and improved.

 

Michael

 

 

Reply