Skip to main content
  • EN

Hey there,

 

I have a few workspaces that make use of the Esri File Geodatabases readers and writers. Processing a huge amount of data comes with some issues. In the past i had noticed a huge slowdown of my performance once i hit the last few transformers in the line.

 

Last week i noticed the log line that comes up during the negative performance switch.

 

The line states: Optimizing Memory Usage (something something)

Ram usage stabilizes from ~2,7 GB down to about 2,4GB and does not rise anymore. I have more than 10GB of available space left but it will not use more, probably because its the 32bit version.

 

After that point, i will only be able to process around 5-10 objects a second! This is extremely slow if compared to before that point, which was around the thousands a second or even more.

 

I am looking for a solution to this issue and came up with the following idea:

If i export the data from the file geodatabase in FME 32bit to something supported by FME 64bit, do the processing on that exported date & after that processing is done, convert it back to esri geodatabase in the 32bit version of FME. This could potentially work around the 32bit resource limits.

 

What do you think of this idea? And to avoid data loss or unwanted mutations, what file formats would be recommended for the export/exchange files between FME 32 and FME 64 bit (while working with file geodatabases).

 

I hope this is a solution many can benefit from. Perhaps even be integrated by the FME application itself?

 

Nick

Hi Nick,

 

As far as I know there shouldn't be a difference for the ESRI file geodatabases that are created using FME 32bit or 64bit if you are using the ESRI (File Geodb *.gdb) reader/writer. Other than Background Processing installation if using ArcMap or ArcPro installed for licensing.

 

Is the issue due to the need for converting ESRI (Personal Geodb *.mdb) geodatabases?

Hi Nick,

 

As far as I know there shouldn't be a difference for the ESRI file geodatabases that are created using FME 32bit or 64bit if you are using the ESRI (File Geodb *.gdb) reader/writer. Other than Background Processing installation if using ArcMap or ArcPro installed for licensing.  

 

Is the issue due to the  need for converting ESRI (Personal Geodb *.mdb) geodatabases?

Thank you for the quick response!

 

I assumed that FME 64 bit is not compatible with ArcGIS 32 bit when it comes to file geodatabases because i would get this error when i try to run it:

Unable to perform any licensing related tasks.  This 64-bit version of FME requires 64-bit Background Geoprocessing, ArcGIS Pro or ArcGIS Server. Refer to this this article http://fme.ly/qua for more details 

According to a website from safe it should be compatible. I think i know understand why; I am still missing the depency called "Background Geoprocessing (64-bit)" from Esri.

 

I'll try to install that and let you know if this the solution i am looking for.

Thank you for the quick response!

 

I assumed that FME 64 bit is not compatible with ArcGIS 32 bit when it comes to file geodatabases because i would get this error when i try to run it:

Unable to perform any licensing related tasks.  This 64-bit version of FME requires 64-bit Background Geoprocessing, ArcGIS Pro or ArcGIS Server. Refer to this this article http://fme.ly/qua for more details 

According to a website from safe it should be compatible. I think i know understand why; I am still missing the depency called "Background Geoprocessing (64-bit)" from Esri.

 

I'll try to install that and let you know if this the solution i am looking for.

No problem! I've gone through this same issue before, so that should fix the licensing issue after you install the  64-bit Background Geoprocessing. Just make sure the background processing version matches the same version of ArcMap that you have installed. If you have ArcPro installed, it should work right out of the box as it's architecture is 64-bit.

Thank you for the quick response!

 

I assumed that FME 64 bit is not compatible with ArcGIS 32 bit when it comes to file geodatabases because i would get this error when i try to run it:

Unable to perform any licensing related tasks.  This 64-bit version of FME requires 64-bit Background Geoprocessing, ArcGIS Pro or ArcGIS Server. Refer to this this article http://fme.ly/qua for more details 

According to a website from safe it should be compatible. I think i know understand why; I am still missing the depency called "Background Geoprocessing (64-bit)" from Esri.

 

I'll try to install that and let you know if this the solution i am looking for.

Try the File Geodatabase Open API reader/writer, it works on 64-bit and does not require the ArcGIS background geoprocessing 

Hi @nick-j​ 

Few options that I can suggest

  1. For 32bit -- Enable 3GB switch for windows 7 -- These instructions should work for windows 10 as well
  2. Enable 64bit Background geoprocessing -- Esri have a small utility which can enable ArcGIS Desktop 32bit to work with FME 64bit
  3. As suggested above, you can use File Geodatabase Open API as well

How to Download ArcGIS Desktop Background Geo processing (64bit) from MyEsri 

  • Log in to MyEsri https://my.esri.com/
  • Click My Organizations --> Downloads 
  • Under Download, Select ArcGIS Desktop Version you have installed
  • And finally download the Utility, here is screen capture from our Downloads page EsriGeoprocessing64bit
  • Go to ArcCatalog / ArcMap --> Help --> About (see below) Catalog_about 

 

Hi @nick-j​ 

Few options that I can suggest

  1. For 32bit -- Enable 3GB switch for windows 7 -- These instructions should work for windows 10 as well
  2. Enable 64bit Background geoprocessing -- Esri have a small utility which can enable ArcGIS Desktop 32bit to work with FME 64bit
  3. As suggested above, you can use File Geodatabase Open API as well

How to Download ArcGIS Desktop Background Geo processing (64bit) from MyEsri 

  • Log in to MyEsri https://my.esri.com/
  • Click My Organizations --> Downloads 
  • Under Download, Select ArcGIS Desktop Version you have installed
  • And finally download the Utility, here is screen capture from our Downloads page EsriGeoprocessing64bit
  • Go to ArcCatalog / ArcMap --> Help --> About (see below) Catalog_about 

 

Thank you for all the information. That sums everything up indeed.

 

The main problem has been tackled, i was missing the background geoprocessing software. I have installed it on my machines with arcgis 10.6.1 and it seems to be running quite good now.

 

One thing i noticed on My Esri, there is no background geoprocessing software for arcgis 10.4.1 (we can unfortunately not upgrade these because our plugins depend on that version alone). It seems that we can only use background geoprocessing on the machines with arcmap 10.6.1+, can you confirm if this is true?

 

I am currently installing background geoprocessing along with arcmap 10.6.1 and FME 2021. This seems to be the only combination where my workspace runs without issues (some known bugs in earlier versions of FME restrict my choice's). Compatibility of the workspace is becoming very restricted .

 

I did ran into another issue with 2021 with the clippers (workspace gets stuck / or / not clipping, even though its inputs are both BREPSolids - non multipart), but i think ill solve that by dividing my dataset into smaller datasets and run it several times. Takes a lot of time to test and will try to isolate it first before opening a new post.

 

Nick

Thank you for all the information. That sums everything up indeed.

 

The main problem has been tackled, i was missing the background geoprocessing software. I have installed it on my machines with arcgis 10.6.1 and it seems to be running quite good now.

 

One thing i noticed on My Esri, there is no background geoprocessing software for arcgis 10.4.1 (we can unfortunately not upgrade these because our plugins depend on that version alone). It seems that we can only use background geoprocessing on the machines with arcmap 10.6.1+, can you confirm if this is true?

 

I am currently installing background geoprocessing along with arcmap 10.6.1 and FME 2021. This seems to be the only combination where my workspace runs without issues (some known bugs in earlier versions of FME restrict my choice's). Compatibility of the workspace is becoming very restricted .

 

I did ran into another issue with 2021 with the clippers (workspace gets stuck / or / not clipping, even though its inputs are both BREPSolids - non multipart), but i think ill solve that by dividing my dataset into smaller datasets and run it several times. Takes a lot of time to test and will try to isolate it first before opening a new post.

 

Nick

@nick-j​  One thing i noticed on My Esri, there is no background geoprocessing software for arcgis 10.4.1 (we can unfortunately not upgrade these because our plugins depend on that version alone). It seems that we can only use background geoprocessing on the machines with arcmap 10.6.1+, can you confirm if this is true?

This is NOT TRUE, or at-least I can download and install ArcGIS for Desktop 10.4.1 version of the software(from My Esri). I did notice that there are 2 product categories ArcGIS Desktop and ArcGIS for Desktop. I would suggest to contact Esri directly and they should be able to give access to installers.

 

For the issues relating BREPSolids -- If you can send us a repro package we can investigate that further. One thing that I would advice is to TURN OFF / Un-check feature caching (Under Run menu).

Thank you for all the information. That sums everything up indeed.

 

The main problem has been tackled, i was missing the background geoprocessing software. I have installed it on my machines with arcgis 10.6.1 and it seems to be running quite good now.

 

One thing i noticed on My Esri, there is no background geoprocessing software for arcgis 10.4.1 (we can unfortunately not upgrade these because our plugins depend on that version alone). It seems that we can only use background geoprocessing on the machines with arcmap 10.6.1+, can you confirm if this is true?

 

I am currently installing background geoprocessing along with arcmap 10.6.1 and FME 2021. This seems to be the only combination where my workspace runs without issues (some known bugs in earlier versions of FME restrict my choice's). Compatibility of the workspace is becoming very restricted .

 

I did ran into another issue with 2021 with the clippers (workspace gets stuck / or / not clipping, even though its inputs are both BREPSolids - non multipart), but i think ill solve that by dividing my dataset into smaller datasets and run it several times. Takes a lot of time to test and will try to isolate it first before opening a new post.

 

Nick

I will do that. Thank you for the suggestions.

Reply


Community Stats

30,364
Posts
115,373
Replies
38,142
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.