I am having issues dissolving large sets of data, the Dissolver works fine and achieves the results I would expect but when I increase the number of data going in the process becomes painfully slow and often results in a crash because of memory.
Page 1 / 1
Hi,
Is it possible to divide your input dataset into different (spatially distinct) subsets? Running the dissolver per subset should increase the performance. If needed you can dissolve the results of the individual subsets in a second step.
Are you using 64-bit FME or 32-bit? Use 64-bit if you can to take advantage of all the memory resources on your system. If the log shows the line "ResourceManager: Optimizing Memory Usage. Please wait..." then you have pretty much used all the memory and FME is now caching to disk.
Make sure the temporary folder points to the faster disk on your system (preferably a solid state drive) and try not to point it to the same disk the operating system is installed on.
If you can post any of the log here it would be interesting to see the section that starts out "System Status" and the lines with "FME Configuration".
For more info check out the chapter on performance in the advanced training manual.
@Mark2AtSafe Having to write to ESRI database so using 32-Bit. at the moment, no ssd unfortunately. I'm not the most IT aware user. If I use to a 64-bit to produce a gml can I then change that back to a esri gdb?
@kd I am working on dividing up at moment.
@Mark2AtSafe Using 64-bit, unfortunately no ssd. I am not the most IT literate FME user.
Trying to dissolve a lot of features across the that have been split by grid lines and although a lot do not need to go through the dissolve I cannot think of a way of separating the ones that do. For example one wood may be split into a max of 3 parts where another a few kms away may be complete. Trying to do this for England and Wales!
I experienced performance issues with Dissolver recently; it helped a lot to precede Dissolver with AreaOnAreaOverlayer and set the Dissolver parameter Assume Input is Clean to 'Yes', and I used parallel processing as I had a GroupBy attribute, but even so the process took hours whereas ArcGIS Pro Dissolve and multipart output did the job in less than 15 minutes.
@Mark2AtSafe Having to write to ESRI database so using 32-Bit. at the moment, no ssd unfortunately. I'm not the most IT aware user. If I use to a 64-bit to produce a gml can I then change that back to a esri gdb?
@kd I am working on dividing up at moment.
I had a client last year who wanted to buffer a road network and dissolve the results and we ran into this problem. The solution I came up with was to use rasters as an intermediate step, this was a lot faster and gave us all the data that the Dissolver was missing. The accuracy of the results depends on the size of the raster cells you use though. Happy to provide sample workspaces.
@Mark2AtSafe Using 64-bit, unfortunately no ssd. I am not the most IT literate FME user.
Trying to dissolve a lot of features across the that have been split by grid lines and although a lot do not need to go through the dissolve I cannot think of a way of separating the ones that do. For example one wood may be split into a max of 3 parts where another a few kms away may be complete. Trying to do this for England and Wales!
https://www.safe.com/transformers/neighbor-finder/
Finally processed and dissolved over 19,000,000 features using 64bit, a combination of some IT tweaks and using FFS, then processed again to use with 32bit. Cheers for help.
Finally processed and dissolved over 19,000,000 features using 64bit, a combination of some IT tweaks and using FFS, then processed again to use with 32bit. Cheers for help.